The collection of images is organized in folders (cat, dog, wild), the
kaggle source has a training and validation folder. The images are
majority clean headshots of cats, dogs, foxes, wolfs, tigers, lions,
other felines, in jpg format.
img.png
img_2.png
img_3.png
The dataset provided contained only training and validation folders,
here is where I have made a slight modification, by creating a test
folder with 500 each images moved from train.
img_4.png
img_5.png
img_6.png
Kaggle source Local Setup Final Local Dataset
Goals
Train and evaluate 3 deep models (EfficientNet, ViT, DenseNet)
Other adjacent objectives:
Train a Fourth model
Run predictions on the test dataset
Tune of parameters, grid search, gather graphs and confusion matrix
DenseNet
What it is, how it works.
In traditional neural networks each layer only receives input from the
previous layer. Example diagram below.
img_7.png
Densely Connected Convolutional Network (DenseNet) is a deep learning
architecture where each layer gets input from all preceding layers,
designed for image classification and other computer vision tasks.
Example diagram below.
img_8.png
This design provides multiple benefits: fewer parameters, greater
computational efficiency, and enhanced generalization.
Our model variant is DenseNet121, the 121 part comes form the fact that
it has a depth of 121(layers).
In our first iteration and practice runs, there was used the pretrained
model. But for the following presentation and charts, we have a
untrained model in all scenarios.
img_9.png
img_10.png
System, parameters and training
Model training was done on a gaming laptop with the following
configuration
img_11.png
In the DenseNet_train.py file, I have added system probing functions
with the purpose to gain knowledge on the system's capabilities.
Example: Is Gpu available, Gpu's limit for batch sizes test, number of
workers test and others. Not all are presented in the final code.
HyperParameters
Our model uses the following hyperparameters:
[Learning Rate (lr)]{.underline} -- Controls how fast the model
updates its weights during training.
[Batch Size (batch_size)]{.underline} -- Determines how many
training samples are processed in one iteration.
[Weight Decay (weight_decay)]{.underline} -- A regularization
technique that prevents overfitting by penalizing large weight
values, ensuring smoother and more generalizable models.
[Optimizer (optimizer)]{.underline} -- Defines how weights are
updated based on gradients. AdamW, used in our case.
[Learning Rate Scheduler (scheduler)]{.underline} -- Dynamically
adjusts the learning rate during training to maintain stable
convergence and avoid premature stagnation.
[Epochs (epochs)]{.underline} -- The number of times the model goes
through the entire dataset. More epochs typically improve learning
but can lead to overfitting if too high.
Hyperparameter Tuning, Grid Search, and Performance Evaluation
img_12.png
These hyperparameters are tested to find the best combination.
After we define our hyper parameter grid, we reach make our first
optimisation in the data loader. Specifically we resize the image to
128x128 since it is provided as 512x512. Our system can not process
efficiently those sizes so in our transformations we perform this
resize. Other transformations to be mentioned, is that we normalize the
images by the mean and standard deviation.
We normalize by scaling pixel values from [0, 255] = [0, 1] to the
pythorch [C, H, W] format.
img_13.png
As next step we use only the training and validation dataset and keep
the test only for prediction testing.
And begin the training to find the best combination of hyperparameters.
img_14.png
Grid contains 3 lr, 3 batch_sizes, 2 weight_decays. Means 3 x 3 x 2 = 18
Total combinations
Total training duration over 2.5h
Other hyperparameters
img_15.png
Adam optimizer with weight decay regularization
A learning rate scheduler that reduces the learning rate after a set
number of epochs
GradScales is an automatic mixed precision (AMP) tool in PyTorch. It
reduces memory usage by using FP16 (half-precision) floating-point
calculations where possible.
img_16.png
We provided the batch sizes 32, 64, 128 and the learning rates 0.001,
0.0005, 0.0001.
Observations:
Smaller batch sizes (32, 64) seem to generalize better, while larger
batch sizes (128) may lead to overfitting or instability.
LR=0.0005 appears to be the most robust learning rate, as it
maintains high accuracy across all batch sizes.
img_17.png
Other generated charts based on our csv.
img_18.png
img_19.png
img_20.png
img_21.png
Prediction
img_22.png
img_23.png
Test Folder
===== test =====
cat = 566
dog = 525
wild = 525
Total in test = 1616
Also found cat image that model predicted to be a dog 😊
img_24.png
Other charts
img_25.png
img_26.png
img_27.png
img_28.png
ViT
What it is, how it works.
ViT (Vision Transformer) is a deep learning model designed for image
recognition tasks, it applies Transformer architectures (originally
designed for NLP) to images.
Works by spliting the image into fixed-size patches (e.g., 16x16 or
32x32 pixels), then each patch is flattened into a 1D vector and
projected into a higher-dimensional space using a linear transformation.
Example diagram below.
img_29.png
Our model variant is vit_b_16, the 16 part comes form the fact that it
patches the image into 16x16 and the base version has a depth of 86M
parameters.
img_30.png
This design provides multiple benefits: performs well on large datasets,
better than CNNs on complex images (once trained properly).
The challenge on a untrained model is that it requires multiple epochs
to achive better accuracy and a large dataset. Also a different strategy
for hyperparameters.
Since the image sizes will me double, 224 instead of the 128 as before
we need to reduce the batch size for faster processing.
8 and 32 batch_size
Pretrained models likely will perform very well no matter the size of
the epochs but we will adjust the following:
From 3 to 4 epochs
On the scheduler, adjust the steps to 1 and the gamma to 0.25, meaning
that it will learn/adjust on each epoch.
img_31.png
Because of the larger image sizes and system performace, we will use a
reduced hyperparameter grid. This will allow us to spot the issues, tune
the parameters and restart if needed.
img_32.png
Observations made:
img_33.png
Model starts with a low accuracy as expected (55%), but improves by 8%,
7%, 5% per epoch. Progression indicates fast gains early, diminishing
returns later.
img_34.png
We provided the batch sizes 8, 32 and the learning rates 3e-05, 0.0001.
img_35.png
Prediction
img_36.png
Other charts
img_37.png
img_38.png
img_39.png
img_40.png
img_41.png
EfficientNet
What it is, how it works.
EfficientNet is a family of convolutional neural networks (CNNs). It is
designed to achieve high accuracy while being computationally efficient.
Unlike traditional CNN architectures, which are scaled manually by
increasing depth (more layers), width (more channels), or input
resolution (larger images), EfficientNet introduces a unique scaling
method called "Compound Scaling" that balances all three dimensions
efficiently.
Example diagram below.
img_42.png
Our model variant is EfficientNet-B0, the B0 part comes from being the
smallest base version and has a depth of 5.3M parameters.
img_43.png
This design provides multiple benefits: Outperforms ResNet, Inception,
and DenseNet with fewer parameters, EfficientNet achieves higher
accuracy with lower computational cost.
Will revert to the image sizes of 128 as before and the previous batch
sizes for faster processing.
32 and 128 batch_size
Pretrained models likely will perform very well no matter the size of
the epochs but we will adjust the following:
3 epochs
On the scheduler, adjust the steps to 1 and the gamma to 0.15, meaning
that it will learn/adjust on each epoch.
img_44.png
Learning rates will be very different, such that we will see the large
variations in the results.
img_45.png
img_46.png
Model starts with a high accuracy as expected (71%), but improves up to
90% .
We provided the batch sizes 32, 128 and the learning rates 0.001,
0.0005.
