Instagram has over 1 billion monthly active users and the backend is powered by Python web servers. Making every server count is a big challenge when the development team becomes large and the code base grows quickly. In this talk, we focus on how I/O latency impacts overall system capacity and reliability, and how we’ve removed latency as the system bottleneck with Python 3’s asyncio module.
Traditional Python web servers process one request at a time using a WSGI interface between the web server and application. This fundamentally puts I/O latency at odds with CPU utilization. Over the last couple of years, Instagram has changed its processing model from single request to concurrent multiple requests, allowing the CPU to process other user requests when long I/O latency is present, unlocking the CPU potential to be adaptable to varied traffic characteristics. In some of our more latency bound data centers or services, we have seen 30+% capacity increase with this model. The concurrent model also brings a set of new challenges: request context vs global state, failure handling, load balancing, stack migration, and performance. We will share some of our learnings and pain points with the community.