PyCon 2012

Advanced Celery

Ask Solem Hoel
Type:
Talk
Audience level:
Experienced
Category:
Distributed Computing
March 10th 2:55 p.m. – 3:40 p.m.

Description

This talk will delve deep into advanced aspects of the Celery task queue and ecosystem. Previous experience with task queues and message oriented middleware is beneficial.

Abstract

  • Tasks

We will look at task examples and rewrite them to better fit the distributed paradigms.

  • Celery + Eventlet

  • Task Routing

    • direct/broadcast/topic

  • Optimization techniques.

  • Monitoring and troubleshooting.

    • Logging (syslog, sentry, error-emails).
    • Events.
    • Tracing memory leaks.

  • Writing a Celery worker in Ruby using celeryd as a proxy.

  • RabbitMQ

    • Clustering and HA

  • Introducing Cyme

  • Q/A