pycon logo

PyCon 2011 Atlanta

March 9th–17th

Log in or Sign Up

Running ultra large telescopes in Python

log in to bookmark this presentaton

Novice / Talk
March 13th 11:55 a.m. – 12:25 p.m.
In this talk I would like to present the SKA (Square Kilometer Array) project in South Africa and how is it run. The main focus will be what the problem area is, how to handle high data-throughput in the telescope and how to make it all work in Python.

Abstract

Running large scale projects is always a challenge. At SKA South Africa, it was decided some time ago that the expressiveness of Python outweighs its potential drawbacks. In this talk I'll present the problem domain of running a large telescope, with a special focus of how to handle it in Python.

Almost everything in this project is written in Python. The production servers run Python for monitoring, control and data processing. Our full testing infrastructure (e.g. simulators) are written in Python. Python technologies in use include NumPy, SciPy and h5py.

On of the central points of our implementation is the KATCP network protocol, which is built in a way that it's possible to introspect devices around you. It has implementations using a normal select loop, twisted, Ruby and C. I'll describe how this protocol works, why it's better than others available and how we manage a large set of devices almost purely within Python.

Lessons learned while implementing software for SKA could be used for any other large-scale high-complexity python scientific projects.

The SKA will be a two thousand antenna radio telescope with an effective collecting area of one square kilometer [1]. Two countries are bidding to host this astronomical instrument -- South Africa and Australia. Construction of the SKA is expected to start in 2016 and be completed around 2024.

South Africa's bid includes the construction of a sixty-four antenna array named MeerKAT which will itself be a leading radio astronomy instrument until the SKA is completed [2]. KAT7, a seven antenna prototype of MeerKAT is already in place on the MeerKAT site.

Many important parts of the KAT7 system use Python, including::

  • monitoring of the entire system
  • simulators for the entire system
  • all low-level devices except the correlator and antenna motor controller
  • control of antenna positions
  • control of all ancillary devices
  • generation and presentation of data for user displays
  • post-processing of data
  • analysis of data (via NumPy, SciPy and CASA)

A number of Python libraries have been released by the project under open source licenses::

  • PySPEAD (the correlator data transfer protocol) [3]
  • KATCP (the monitoring and control protocol) [4]
  • an HTML5 backend for matplotlib [5]
  • CASPER correlator interface utilities [6]

[1] http://en.wikipedia.org/wiki/Square_Kilometre_Array [2] http://en.wikipedia.org/wiki/MeerKAT [3] http://github.com/sratcliffe/PySPEAD [4] http://pypi.python.org/pypi/katcp/0.3.0 [5] http://code.google.com/p/mplh5canvas/ [6] http://pypi.python.org/pypi/corr/0.5.1