Jupyter Notebooks: the anti-pattern of good engineering

The use of Jupyter Notebooks as a quick scratchpad for prototyping or developing Exploratory Data Analysis reports is the slippery slope that will lead your team to a bunch of bad software engineering practices.

The Notebook appeal

Jupyter notebooks got extremely popular in data work, because they're easy to use, interactive, and can combine code and markdown so you can produce reports on the spot. That's very understandable but due to the fast-paced work environments, and the iterative nature of data work, many teams have started misusing it and drifting into bad software engineering habits. By good practices I mean good modularization, logging, testing, and good documentation.

Code is code

Many teams don't follow these practices either

• Because they don't know that they need to, or they don't know how to apply them
• Because they think they're only applicable on shippable software products, and that they don't need to apply them in data analytics context

But that's very problematic because at the end of the day, it's still code. And even if you're developing a one-off report on some unique data, it's still a development process if you consider the report your final artifact, you'd still iterate, and run into bugs. These practices aren't decorative, and we don't just follow them because some academic decided that they are useful!

So, are notebooks useless?

Absolutely not, you just need to leverage them correctly! Notebooks should be your presentation layer, not your development environment. All your logic should live outside the notebook, abstracted, isolated, tested, and your notebook should just import from your modules and execute to present the results.

A good structuring of an exploratory data analysis project should look something like this:

• Data extracting module/package one module is enough if you're dealing with one type of data source, but if you're extracting from a database, calling an API, reading files and scraping the web, isolate each in a separate file
• Data cleaning/wrangling module
• Data preprocessing and feature extraction module
• Model training module
• Evaluation module
• Visualization module
• A Notebook that imports the relevant functions from these modules, and calls them in cells surrounded by other markdown cells

You can even create a template repository with this structure containing abstracted logic, clone it for every new EDA project, use whatever useful existing functions you can utilize and only write new functions when you need new functionality. Even that unique dataset you're working on could use the same standard cleaning you give to familiar datasets. Follow DRY (Don't Repeat Yourself) and reuse functions that strips strings, parse dates, or interpolate/drop nulls instead of rewriting them every time.

Take it further

This setup, not only allows you to prototype quickly through templating, or debug easily through proper modularization and isolation, but also allows you to take your next step in DataOps. If you have frequently requested report feeding on the same data sources, you can automate that by adding a validation layer to ensure the upstream data is still following the same format and use parameterized notebooks, and create your own pipeline that picks up the fresh data, feeds it to your notebook, executes the notebook and save its resulting artifact as a PDF or any other format.