When we think about what Python is for, we often think of things like analytics (SciPy, NumPy, Pandas), machine learning (scikit-learn, PyTorch, Keras), and web apps (Django, Flask). But Python (together with its rich ecosystem of open source libraries) plays a tremendous and often invisible role in our everyday lives, from medicine to finance, and even the very electricity that powers our cities. In fact, the power grid is one of the most interesting domains for Python software implementations both because of the massive amount of data it generates, and the rather high stakes nature of execution.
To cope with increasing load and a shift toward demand-side generation, today’s electric grids are increasingly monitored by a fleet of networked sensors. Together, these sensors provide control room operators with real-time snapshots of the grid that allow them to bolster physical protection mechanisms with predictive maintenance and rapid response. However, they also generate petabytes of data that must be analyzed in real-time to detect microsecond events across a wide area, often with variable connectivity that leads to poor data quality and makes traditional time series analytics very difficult.
Flexibility is the key to merging analytics that use sliding window analyses such as voltage sag detection with jitter-prone, variable sensor streams, which makes Python the ideal tool. In this talk, we will describe using a time-partitioned, measurement aggregating tree structure with btrdb-python to facilitate rapid queries on dense telemetry, using power engineering use cases a guide. We will also describe the design and development of conduit, a Python data flow library that interacts with btrdb-python to allow descriptive preprocessing of data on demand.