PyCon 2016 in Portland, Or
hills next to breadcrumb illustration

Joy and Confidence in High Profile Software: A Test Harness for Climate Data Algorithm Development

Ryan Berkheimer

Audience level:


Global climate research is now under intense public scrutiny. The results from the algorithms composing this research are all painstakingly peer reviewed, but the original fortran code can seem obtuse. While NOAA engineers refactor the code, a reusable, portable test harness written in python, using mongo and js, makes testing and progress tracking easy and transparent to all.


With a national spotlight now brightly shining on climate research, algorithms that have been in use for 30 years or more are under increasing scrutiny. For example, the pairwise homogeneity algorithm, which automatically corrects temperature data for faulty field instrument response functions, was recently debated in a Washington Post article, gathering over 1100 public comments. Because of this scrutiny, the public has rightfully demanded greater access to the source code itself. Mostly developed in Fortran 77, with heavy use of awk, bash, and make, the code was written by the physical scientists during development. The code is designed to be efficient to the extreme and scientifically accurate, but this often makes it appear daunting and obtuse. NOAA has undertaken the obligation to make these algorithms accessible and transparent to the public, while maintaining the peer reviewed integrity of the results, and a team of engineers is currently performing the refactor on the pairwise homogeneity algorithm (PHA). This effort is being made in modern fortran. Partly because of the limitations of the fortran language, and partly due to other motivating factors of scientific integrity, stakeholder concern, and enhancing public transparency, there was a great motivation to create a test harness in another language. With f2py, a great mongo driver, functional style capability, and easy web frameworks, python was a natural choice as the language for a test harness. The test harness was designed in concert with a generic entity REST based storage in mongo. The storage system allows for a customizable definition of model templates with user defined properties and arbitrary nesting. For the PHA project the input data storage pattern is a Station model, which is a container for both a Location model and a Data model that houses various containers, each eventually branching into a Measurement model. A dataset can be loaded into a defined model and split into records (model instances). These records are then trackable over time - once loaded, a record is only mutable in the sense that its end date is modifiable. This is important, as it allows comparisons to be made on identical input datasets at any point in time. The test harness also uses a server model, which contains information on the compiler, file paths, etc, a DataRun model, which contains information on the version of the software version itself, input parameters, etc, and a comparison model, which allows comparison of DataRuns. To use the test harness, the user is presented with a web-based GIS that displays a set of stations available. The user then is able to make a choice of what stations to use, which dataset to use, and what endpoint step to run (what function in the PHA to endpoint). The user then runs the software, which makes appropriate calls to the server and creates a DataRun, producing and parsing output, which is then made available to the user for front end viewing. Two DataRuns that have the same input can then be compared, which produces a data comparison. The results of the comparison and each DataRun is all stored in another mongo collection. By using modern big data technologies, malleable schema storage, and a responsive and descriptive visual front end, the rejuvenation of difficult, seemingly complex, and unapproachable algorithms can be made confidently and even joyfully, and python is really the only realistic choice for such a task.