Pytorch library¶

The most popular Python ML and deep learning library to implement ML workflow and deep learning solution. It is open-source project. It helps to run code on GPU/TPU. PyTorch is also a low-level math library as NumPy, but built for deep learning. It compiles these compute graphs into highly efficient C++/CUDA code.

The sources for this content is from product documentation, Zero to mastery - learning pytorch, and WashU training website.

Environment setup¶

Use pip or mini conda for package management and virtual environment management, and jupyter notebooks.

Install¶

• Using Python 3 and pip3, use a virtual environment, install torch

  pip3 install torch torchvision torchaudio
  

• Using Anaconda:

  1. Install miniconda (it is installed in ~/miniconda3):

     # under ~/bin
     curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh
     sh Miniconda3-latest-MacOSX-arm64.sh -u
     

  2. Verify installed libraries: conda list

  3. Environments are created under ~/miniconda3/envs. To create a conda environment named "torch", in miniconda3 folder do: conda create anaconda python=3 -n torch
  4. To activate conda environment: conda activate torch
  5. Install pytorch conda install pandas pytorch::pytorch torchvision torchaudio -c pytorch
  6. [optional] Install jupyter packaging: conda install -y jupyter
  7. Register a new runtime env for jupyter: python -m ipykernel install --user --name pytorch --display-name "Python 3.11 (pytorch)"

Run once conda installed¶

1. To activate conda environment: conda activate torch
2. Test my first program: python basic-torch.py
3. If we need Jupyter: jupyter notebook in the torch env, and then http://localhost:8888/tree.
4. Select the Kernel to be "Python 3.9 (pytorch)"

My code studies are in pytorch folder.

Concepts¶

Tensor¶

Tensor is an important concept for deep learning. It is the numerical representation of data, a n dimension matrix.

Tensors are a specialized data structure, similar to NumPy’s ndarrays, except that tensors can run on GPUs.

matrix1 = torch.tensor([[1, 1, 1], [1, 1, 2]], device=device, dtype=torch.float16)

Tensor attributes describe their shape, datatype, and the device on which they are stored.

import torch, numpy as np
shape = (2,3,)
rand_tensor = torch.rand(shape)
ones_tensor = torch.ones(shape)

Tensor created from NumPy array:

import torch
X= torch.from_numpy(X).type(torch.float)
y= torch.from_numpy(y).type(torch.float)
X[:5],y[:5]

Tensors on the CPU and NumPy arrays can share their underlying memory locations, and changing one will change the other. See the set of basic operations on tensor.

See the basic ML workflow using Pytorch to work on data and do a linear regression workflow-basic.ipynb.

Constructs¶

PyTorch has two important modules we can use to create neural network: torch.nn, torch.optim, and two primitives to work with data: torch.utils.data.DataLoader and torch.utils.data.Dataset. Dataset stores the samples and their corresponding labels, and DataLoader wraps an iterable around the Dataset. See dataset examples

• See basic code in basic-torch.py with inline explanations.

GPU¶

On Linux or Windows with nvidia GPU, we need to use the Cuda (Compute Unified Device Architecture) library. See AWS deep learning container. For Mac, use mps.

Here is sample code to set mps to access GPU (on Mac) for tensor computation:

has_mps = torch.backends.mps.is_built()
device = "mps" if has_mps else "cuda" if torch.cuda.is_available() else "cpu"

NumPy uses only CPU, so we can move to tensor and then tensor.to(device) to move the tensor to GPU, do computation and move back to NumPy

tensor=torch.tensor([1,2,3])
tensor_on_gpu = tensor.to(device)
tensor_back_on_cpu = tensor_on_gpu.cpu().numpy()

See Tim Dettmers's guide.

Basic Algebra with Pytorch¶

See Algebra using Pytorch python code.

Loss functions¶

Cost / loss functions selection depends on the problem to solve.

The binary cross-entropy / log loss is used to compute how good are the predicted probabilities. The function uses a negative log probability for a label to be one of the expected class: {0,1}, so when a class is not 1 the loss function result is big.

Neural network¶

A PyTorch neural network declaration is a class that extends nn.Module. The constructor includes the neural network structure, and the class must implement the forward(x) function to pass the input to the network and get the output. The back propagation is done using SGD. This is the most flexible way to declare a NN. As an alternate the following code uses the Sequential method using the non linear (nn.ReLu()) function between layers.

model = nn.Sequential(
    nn.Linear(x.shape[1], 50),
    nn.ReLU(),
    nn.Linear(50, 25),
    nn.ReLU(),
    nn.Linear(25, 1)
).to(device)

Neural network has an input layer with # of parameters equal to the number of input features, and the number of output equal to the number of expected responses (1 output for binary classification). The first layer above, is a linear transformation to the incoming data (x):

50 is the number of parameters to the first hidden layer. For activation function between hidden layers, ReLU (max(0,x)) is often used when we want non-linearity. The output layer will not use a transfer function for a regression neural network, or use the logistic function for binary classification (just two classes) or log SoftMax for two or more classes.

The hyper-parameters to tune are:

• The number of neuron in hidden layer: In general, more hidden neurons means more capability to fit complex problems. But too many, will lead to over fitting. Too few, may lead to under fitting the problem and will sacrifice accuracy.

• The number of layers: more layers allow the neural network to perform more of its feature engineering and data preprocessing.

• The activation function between hidden layers and for the output layer.
• The loss and optimizer functions.
• The learning rate of the optimization functions
• Number of epochs to train the model. An epoch as one complete pass over the training set.

For multi class training, LogSoftmax is used as transfer function and CrossEntropyLoss as loss function. With Softmax, the outputs are normalized probabilities that sum up to one.

Some code samples:

• Basic NN in dual class classifier notebook to identify plots on 2 circles.
• Multi-class classifier notebook
• Python code for a PyTorch neural network for a binary classification on (Sklearn moons dataset) using Loss : nn-classifier.py.
• Computer vision and the CNN A notebook and Python code

{'model_name': 'FashionMNISTModel', 'model_loss': 0.41334256529808044, 'model_acc': tensor(0.8498, device='mps:0')}
{'model_name': 'FashionNISTCNN', 'model_loss': 0.3709910213947296, 'model_acc': tensor(0.8716, device='mps:0')}

Model training¶

PyTorch training loop¶

For the training loop, the steps to build:

Example of code for training on multiple epochs:

loss_fn=nn.CrossEntropyLoss()
optimizer= torch.optim.SGD(params=model.parameters(), lr=0.1)

for epoch in range(epochs):
    model.train()
    # 1. Forward pass
    y_logits = model(X_train).squeeze()
    # from logits -> prediction probabilities -> prediction labels
    y_pred = torch.softmax(y_logits,dim=1).argmax(dim=1)
    loss = loss_fn(y_logits,y_train)
    acc = accuracy_fn(y_true=y_train, y_pred=y_pred)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

The rules for performing inference with PyTorch models:

model.eval()

# 2. Use the inference mode context manager to make predictions
with torch.inference_mode():
    y_preds = model(X_test)

See train_step function in engine.py

PyTorch testing loop¶

The typical steps include:

model.eval()
with torch.inference_mode():
    # 1. Forward pass
    test_logits = model(X_test).squeeze() 
    test_pred = torch.softmax(test_logits, dim=1).argmax(dim=1)
    # 2. Caculate loss/accuracy
    test_loss = loss_fn(test_logits,y_test)
    test_acc = accuracy_fn(y_true=y_test, y_pred=test_pred)

See test_step function in engine.py

Improving a model¶

Evaluate classification models¶

Classification model can be measured using the at least the following metrics (see more PyTorch metrics):

Pytorch datasets¶

PyTorch includes many existing functions to load in various custom datasets in the TorchVision, TorchText, TorchAudio and TorchRec domain libraries.

See prepare_image_dataset.py to get food images from PyTorch vision.

Data augmentation¶

Data augmentation is the process of altering the data in such a way that this artificially increases the diversity of the training set.

The purpose of torchvision.transforms is to alter the images in some way and turning them into a tensor, or cropping an image or randomly erasing a portion or randomly rotating them.

Training a model on this artificially altered dataset hopefully results in a model that is capable of better generalization (the patterns it learns are more robust to future unseen examples).

Researches show that random transforms (like transforms.RandAugment() and transforms.TrivialAugmentWide()) generally perform better than hand-picked transforms.

We usually don't perform data augmentation on the test set. The idea of data augmentation is to artificially increase the diversity of the training set to better predict on the testing set.

See also in PyTorch's Illustration of Transforms examples.

Transfer learning for image classification¶

With transfer learning is to take an already well-performing model on a problem-space similar to the one to address and then to customize it.

For custom data to go into the model, need to be prepared in the same way as the original training data that went into the model.

PyTorch models has weights and we can get the transformers from the weight.

weights = torchvision.models.EfficientNet_B0_Weights.DEFAULT
transformer= weights.transforms()
model=torchvision.models.efficientnet_b0(weights=weights).to(device)

efficientnet_b0 comes in three main parts:

• Features: A collection of convolutional layers and other various activation layers to learn a base representation of vision data.
• avgpool: Takes the average of the output of the features layer(s) and turns it into a feature vector.
• classifier: - Turns the feature vector into a vector with the same dimensionality as the number of required output classes (ImageNet has 1000 classes, out_features=1000).

The process of transfer learning usually freezes some base layers of a pre-trained model, typically the features section, and then adjusts the output layers (also called head/classifier layers) to suit our needs.

Some How to¶

def getDevice():
    if torch.backends.mps.is_available():
        device = torch.device("mps")
    elif torch.backends.cuda.is_available():
        device = torch.device("cuda")
    else: 
        device = torch.device("cpu")
    return device   

# saving using Pytorch
MODEL_SAVE_PATH = MODEL_PATH / filename
torch.save(model.state_dict(), MODEL_SAVE_PATH)
# Load is reusing the class declaration
model=FashionNISTCNN(input_shape=1,hidden_units=10,output_shape=10)
model.load_state_dict(torch.load("models/fashion_cnn_model.pth"))

def make_confusion_matrix(pred_tensor, test_labels, class_names):
    # Present a confustion matrix between the predicted labels and the true labels from test data
    cm = MulticlassConfusionMatrix(num_classes=len(class_names))
    cm.update(pred_tensor, test_labels)
    fig,ax = cm.plot(labels=class_names)
    plt.show()

train_transformer=v2.Compose([v2.Resize((224,224)), v2.TrivialAugmentWide(num_magnitude_bins=31), v2.ToTensor()])

import torchinfo
torchinfo.summary()

Code samples¶

• Basic operations on tensor: my own notebook and Learn Pytorch introduction.
• Pytorch workflow for training and testing model
• Compute image classification on Fashion NIST images in pythons and use_fashion_cnn.pn
• Pizza, steak, sushi image classifier

Resources¶

• PyTorch.org tutorial
• Udemy content repo
• Zero to mastery - learning pytorch
• The incredible pytorch: curated list of tutorials, projects, libraries, videos, papers, books..
• Dan Fleisch's video: What's a tensor?
• How to train state of the art models using Torchvision - PyTorch blog.
• Jeff Heaton - Using Convolutional Neural Networks.