Saturday 12:10 p.m.–12:55 p.m.

Subprocess to FFI: Memory, Performance, and Why You Shouldn't Shell Out

Christine Spang

Audience level:
Intermediate
Category:
Python Core (language, stdlib, etc.)

Description

Sometimes, the code you need just isn't available in Python, but someone's written some code in another language that solves your problem. What options do you have for interacting with this external code from within your program? We'll examine how to call external binaries, what happens behind the scenes at a systems level, and how these implementation details affect performance and memory usage. In the second part of the talk, we'll discuss the different options for more tightly integrating external code in the form of a C library for better performance and control.

Abstract

In this talk we'll explore different options for interacting with non-Python code from Python, starting with invoking an external process using the subprocess module and ending with integrating external C libraries. We'll discuss tradeoffs encountered when building systems along the way. ** "subprocess" behind the scenes ** There's a whole lot going on when you invoke another process with the functions available from the subprocess module. The Python API is simple, but it turns out that there's a whole lot going on underneath the hood when you call out to another process, and a lot of these details are important to debugging and addressing performance issues. We'll deep dive down into what's going on here and learn about the interface between userspace applications and the Linux kernel. ** wrapping C library code ** With C libraries, we have the option of more tightly integrating with a Python program, which offers us more performance and control than shelling out. We'll learn what a Foreign Function Interface is, and take a look at the different options for integrating C libraries, including ctypes, Cython, and cffi. We'll discuss the pros and cons of these options and make some recommendations for which to use when. You'll leave this talk with a better understanding of how Unix processes work, knowledge of the options available for integrating C with Python, and perhaps some inspiration on how to use the tools available to you to gain an understanding of a system when you just don't know what's going on. A reading knowledge of C will be useful but not required.