Executing long-running Jupyter notebooks in the background


I have some notebooks that take over 12 hours to complete and use enough memory that they must be run on a large server. While you could set up VNC server or something to have a persistent, interruptible session, I prefer this short function that I just run in a screen session. It's pretty straightforward, except for --ExecutePreprocessor.timeout=-1 which is necessary since each cell has only 30 seconds to finish running by default.

function runbook() {
    jupyter nbconvert --to notebook --ExecutePreprocessor.timeout=-1 --execute --inplace $1
}

Then it's as simple as running:

runbook my-long-running-notebook.ipynb