Catalyst 101 — Accelerated PyTorch

Break the cycle — use the Catalyst!

PyTorch is great framework to create deep learning models and pipelines. Nevertheless, for all its merits, it could use improvements in terms of writing training loops, validating and testing neural networks. Moreover, PyTorch users are likely to introduce more bugs during the research and development process as they mix in complicated things like mutli-GPU, mixed precision, and distributed training.

For real breakthroughs in deep learning, we need a strong foundation. In this blog post, I would like to introduce Catalyst framework, developed with focus on reproducibility, fast experimentation and code/idea reusing. We’ll also provide a tutorial on MNIST classification problem as an example.

Catalyst introduction

What Catalyst is? What is it for? And who are the maintainers?

Catalyst is PyTorch framework for Deep Learning research and development. You get a training loop with metrics, model checkpointing, advanced logging and distributed training support without the boilerplate.

Catalyst pipeline example — with 30 lines of code you get model training, inference and tracing support

I applied my learnings from working on machine learning projects, both academia and industry, in various fields— CV, NLP, GANs, RL etc. As a result, the framework is designed to be extremely extensible and production-friendly while making state of the art DL research techniques (like distributed training) trivial.

Moreover, Catalyst.Team and Catalyst.Contributors consist of industrial and academia researches, kaggle masters and professional software engineers. You can check out Catalyst-based kaggle competitions, pipelines and projects here (more than 100 production-ready solutions already).

Catalyst CI performs tests under various training conditions

Catalyst is rigorously tested and supports extra subpackages for Computer Vision and Natural Language Processing.

In addition, Catalyst.Team provides several production-ready pipelines,

• catalyst-team/classification — comprehensive classification pipeline
• catalyst-team/segmentation — binary and semantic segmentation pipeline
• catalyst-team/detection — anchor-free detection pipeline
• catalyst-team/video — video classification pipeline
• catalyst-team/bert — DistilBERT pipeline for token classification tasks (archived version, updated release soon)

And extra packages for convenient Deep Learning R&D,

• Alchemy — experiment logging and sharing tool
• Reaction — deep learning models serving with batch handling and autoscaling
• Catalyst.RL — Reinforcement Learning research package, used to win 3rd place at NeurIPS 2018 and 2nd one in 2019.

Getting started

But let’s make it simple, how could you start with Catalyst and join our Community? To begin with, let’s walk through MNIST classifier tutorial and compare PyTorch and Catalyst code side-by-side. While Catalyst supports arbitrarily complicated multi-stage deep learning pipelines, it’s still better to start with the basics.

Typical DL workflow

Each deep learning project typically consists of several main components:

• the data
• the model(s)
• the optimizer(s)
• and the loss(es)

The data

For this tutorial we’re using MNIST.

MNIST

Long story short, both Pytorch and Catalyst have the very same 4 steps for data preparation:

1. Download images
2. Split them into train and validation datasets
3. Add extra image transforms for splits (these are highly subjective)
4. Wrap each dataset split into a DataLoader

Thanks to PyTorch team, 1–3 stages are already implemented for us:

At this step we have no differences between PyTorch and Catalyst code.

The model

Let’s design a 3-layer fully-connected neural network that takes as input an image that is 28x28 and outputs a logits distribution over 10 possible labels.

This model defines the computational graph to take as input an MNIST image and converts it to a logits distribution over 10 classes for digits 0–9.

Same here, we have no additional abstractions or custom modules, pure PyTorch code.

Btw, Catalyst also provides a wide variety of contrib models, for examples,

This way you can get the same model, but it’s a bit more extendable due to hiddens parameter.

The optimizer(s)

Now we choose how we’re going to do the optimization. For now, let’s use Adam as it is a good default in most DL projects.

Quite simple, isn’t it? It is as simple and powerful as PyTorch. Moreover, Catalyst provides a large variety of contrib Optimizers and easily handles multi-optimizer case, for example, for GANs experiments.

Catalyst works on top of PyTorch API and provides large number of extensions and user-friendly Experiment API to deliver “catalyzed” version of PyTorch. This ways you can boost your DL progress onto new level!

The loss(es)

Let’s use typical cross-entropy loss for our multi-class classification problem.

Again… the same code and full power of Catalyst contrib modules!

Training loop

And here we have all main components of our Experiment:

1. The dataset (Mnist)
2. The model (3-layers NN)
3. The optimizer
4. The loss (criterion)

Now, let’s check how typical train loop looks like,

Deep Learning generalized training loop

Long story short, usually we

• Go through our experiment stages (pretraining, training, finetuning)
• Iterate for many epochs
• Divide datasets in each iterated epoch into small samples of data — batches
• Perform model forward pass
• Compute the loss
• Perform a backward pass to calculate all the gradients
• Apply them to the model

And here we come to the Catalyst advantages. In PyTorch, you have to write the for loops yourself which means you have to remember to call the correct things in the right order — thinking more about engineering stuff, rather than your hypothesis testing.

Even if your model is simple, everything becomes complicated once you start doing more advanced things like multi-GPU or distributed training, gradient clipping, half-precision usage, early stopping, model checkpointing, etc…

And here comes the power of Catalyst. It abstracts the boilerplate and leaves main parts to you. You get a training loop with metrics, model checkpointing, advanced logging and distributed training support without the boilerplate. The same PyTorch code but much more readable and fully reproducible (another Catalyst feature).

Training Loop with PyTorch

The full MNIST example written in PyTorch is as follows:

Training Loop with Catalyst

The full MNIST example with Catalyst is exactly the same, but:

• The training/validation loop code has been abstracted by the Runner
• You have specified key point for both training and inference parts
• You get automatic model validation and checkpointing
• You get easy access to PyTorch tracing, distributed and half-precision training
• Finally, you can focus only on key parts of your research

Catalyst extras

With Catalyst.dl and Catalyst.contrib the training loop can be simplified. You get access to

• a variety of predefined metrics
• a full-featured Callback mechanism
• a bunch of Catalyst Loggers (Alchemy, Neptune, Tensorboard, Wandb)
• a large number of contrib modules, supported by Catalyst Community
• Deep Learning best practices, like `load_best_on_end` feature

Conclusion

• Catalyst is pure PyTorch framework without any custom modules.
• With Catalyst you get the same PyTorch pipelines but in much more readable and fully reproducible way.
• Catalyst is easily extendable with Runner and Callback abstractions.
• You focus on key parts of your deep learning R&D.

So, what are you waiting for?

Quick Start

• Introduction
• Minimal examples
• Docs
• Guides

In future tutorial series we will discuss more Catalyst-based examples, best practices and pipelines. Stay tuned!

More than a framework — Ecosystem

Combining human creativity with machine’s computational power

Catalyst.Ecosystem manifesto

Catalyst was designed by researchers for researchers,

• to get the foundation for DL & RL research, which would be well tested and verified
• to generalize and develop best practices that work with all DL areas
• to think less about tech-side and focus on research hypothesis and their value
• to accelerate your research

Catalyst.Ecosystem,

• combines research best practices and helps knowledge sharing
• connects DL/RL with Software Engineer for maximum performance
• allows quickly and efficiently validate your hypotheses

Catalyst main principles,

• open — it’s fully Open Source Ecosystem
• equivalently — everyone can contribute and propose new ideas
• expertise — we are gathering top deep learning knowledge into one place
• high-performance — we are developing the ecosystem with software engineering best practices

Open Source Research Community

Last but not least, Catalyst.Friends community connects different startups, companies and research labs all over the world. For example, Catalyst.Team is working with Translational Research in Neuroimaging and Data Science (TReNDS) on DL brain image analysis. Feel free to write us, if you are also interested in such collaborations.