The recent cloud buzz has largely benefited Python web application developers using web frameworks such as Django. Google AppEngine, for example, allows anybody in the world to easily host their own website. But, for Python's formidable scientific community, the cloud has been noticeably less helpful--until now. PiCloud is a Python-based cloud platform that tackles a noble cause: giving every scientist in the world instant access to a supercomputer. The talk will cover how Python inspired the design of PiCloud, which has now processed over 100M jobs.
PiCloud offers the easiest way to utilize the cloud for compute-intensive applications; more specifically, applications in scientific computing, high-performance computing, and batch processing. With only a couple lines of Python code, scientists, developers, and engineers on our platform can leverage thousands of cores of computational power on-demand.
Our goal is to make computing power a utility for scientists in the same way that electricity is for modern society: available to everyone, seemingly infinite in quantity, and readily accessible with a flip of a switch. We achieve this by being a serverless cloud. In other words, our users each get the power of a supercomputer at their fingertips without having to design, provision, or administer any servers.
PiCloud was started thanks to a serendipitous collision of computer vision and a curiosity for the Python language. In 2009, we were working on a fun application called AutoTagger to automatically tag peoples faces on Facebook. While we coded the project in Python and C++ using extensions, we realized we were spending over half of our time administering our cluster to process thousands of photos in parallel, rather than on our Python vision algorithms. At the same time, being fairly new to Python, we were exploring some of Python's more intricate features: pickling, function and module introspection, variable keyword arguments, and higher-order functions.
With Python's amazing toolbox at our disposable, we realized we could provide a Pythonic way for anyone, but especially scientists, to use an innumerable number of cores without ever having to halt the development of their algorithms to manage servers. The result was the cloud package, which takes advantage of many of Python's introspective features to automatically move computational work to the cloud. Here's a simple example to showcase how easy it is:
import cloud from autotagger.vision import face_detector from autotagger.facebook import get_friends_photos list_of_photos = get_friends_photos() # detect the faces of my friends in parallel on the cloud job_ids = cloud.map(face_detector, list_of_photos)
Two new lines of code unlock the power of potentially thousands of cores on the cloud. We'll also show the a-ha moment where we realized the automagic of PiCloud was possible.
Python's strength is its diversity of users. Thanks to its readability, ease of use, and extensive libraries (NumPy, SciPy, etc...), Python is popular among all scientists, rather than just computer scientists. Likewise, PiCloud's focus on simplicitly caters to those who aren't necessarily programmers by training. We believe accessibility is critical in an age where science has become increasingly reliant on server farms for data analysis.
Examples of what's been done on PiCloud with Python:
We've opened up the accessibility and availability of computing to everyone in the world who doesn't work at a national laboratory, or a Google. It's for this reason that we were credited by a user as "accelerating the speed of science." PiCloud has processed over 100 million jobs for our three thousand users worldwide.
Initially as a two-man team, we were hard-pressed to develop fast and release quickly to validate our idea to ourselves and potential investors. Python was a great choice of language for us because it allowed us to develop fast without sacrificing readability with only modest sacrifices in performance; we made sure that all the heavy lifting was done in Python libraries that pushed worked to C. We're happy to share stories of features that had to be built in 0 days, and were thanks to Python. :)
For audience members looking to start a new project: There are a lot of technology building blocks are readily available and robust that make it incredibly easy to go from concept to product in a short time frame with a small team.
Since PiCloud is built almost entirely in Python for Python, we will share novel portions of our architecture. This section is dedicated to educating the audience on the next-generation of distributed systems that are made possible by the ease of server provisioning on the cloud and Python.
We owe our gratitude to the entire Python community. More than just forming the bedrock of our platform, they've enabled us to change the face of parallel computing and empower thousands of scientists around the world.