Presentation: Python Profiling

Friday 3:15 p.m.–3:45 p.m.

Python Profiling

Amjith Ramanujam

Audience level:: Intermediate
Category:: Useful Libraries

Description

This talk will give a tour of different profiling techniques available for Python applications. We'll cover specific modules in Python for doing function profiling and line level profiling. We'll show the short comings of such mechanisms in production and discuss how to do sampled profiling of specific functions. We'll finish with statistical profilers that use thread stack interrogation.

Abstract

We'll begin with the basics of profiling techniques using the cProfile module in the Python standard library. Cover how to analyze the data using pstats and RunSnakeRun. Some of the functional limitations of the cProfile module will be discussed and we'll see how to get past the limitation by introducing the line_profiler module. These profilers are suited for a development environment due to their high overhead.

We'll discuss some ideas about doing profiling in production. We'll talk about using sampling techniques. That is, only profiling specific functions and only every so often, so we are not profiling whole application all the time. Then also look at separate technique of sampled profiling using thread stack interrogation as opposed to built-in profiling modules which are using Python itself to tell you about all function enter/exit events. This will include some thrid-party tools such as plop and NewRelic's thread-profiler.

Presentation of the tools will include code snippets and demo.