Description
Interactive Notebooks (e.g., Jupyter, Colab) are widely used for data science tasks such as data analysis, data visualization, and model training. A fundamental limitation of vanilla notebook frameworks is the inability to undo the effect of cell executions on the session state, e.g., users cannot ‘un-drop’ dataframe columns. In this talk, I will describe our ongoing work on adding automated, lightweight checkpointing in notebook sessions to address this issue.
Audience
This talk should be interesting for an intermediate-level audience who have some experience with using Python-based notebooks and common data science libraries (e.g., Pandas, Matplotlib) and have experienced first-hand the various issues of existing notebook frameworks (e.g., no undos, non-persistent state). This talk will assume prior knowledge of notebook basics and terminology (e.g., cell executions, session state).
Objectives
Attendees will learn about how incremental checkpointing and versioning techniques commonly utilized in databases can be adapted to work well with interactive notebooks. At a more detailed level, they will learn about the characteristics of typical notebook workloads and how information (e.g., dependencies between variables) required for incremental checkpointing can be profiled from cell executions.