This is a perforated ResNet-18 model, one enhanced with dendritic optimization, a novel technique that increases accuracy with significantly improved parameter efficiency. The model was pretrained on ImageNet and can be used achieve for transfer learning with closer to ResNet-34 level performance while using a fraction of the parameters.
There are two models in this family. resnet-18-perforated-gd has 5 dendrites which were trained with gradient descent. resnet-18-perforated-cascor has 2 dendrites added and trained dendrites with Perforated Backpropagation via the Cascade Correlation learning paradigm.
Experimental results can be seen in the spreadsheet here.
What is Dendritic Optimization?
Dendritic optimization (referred to as "perforation") enhances neural networks by adding specialized dendrite nodes to individual neurons. Each dendrite node receives the same inputs as its parent neuron but provides a dedicated output channel exclusively to that neuron. This allows neurons to make more sophisticated decisions about the features they encode, similar to how biological dendrites help neurons process information.
Think of it as giving each neuron its own set of specialized advisors that help it better understand the features it's responsible for detecting.
Model Details
Architecture: ResNet-18 with perforated pre-FC layer
Parameters: 12.3M (compared to 11.7M for standard ResNet-18, 21.8M for ResNet-34)
Dendrites: 2 dendrite nodes per neuron in the pre-FC layer
Pre-training: ImageNet-1k
Framework: PyTorch
License: Apache 2.0
Architecture Flow
Main ResNet-18 backbone
Pre-FC Layer (Perforated FC layer with 3 dendrites)
FC Layer (replaceable for transfer learning)
The uploaded FC layer has randomized weights, this model is only for use in transfer learning where the FC layer should be replaced.
Performance
Parameter Efficiency on ImageNet
The key advantage of dendritic optimization is dramatic improvement in parameter efficiency. When considering the trade-off of moving from ResNet-18 to ResNet-34:
Model
Parameters
ImageNet Accuracy
Percentage Gain per M Params
ResNet-18
11.7M
69.76%
- (baseline)
ResNet-34
21.8M
73.30%
0.35
ResNet-18-prefc-gd + 1 Dendrites
12.2M
70.99%
2.34
ResNet-18-prefc-gd + 2 Dendrites
12.5M
71.37%
2.03
ResNet-18-prefc-gd + 3 Dendrites
12.7M
71.65%
1.79
ResNet-18-prefc-gd + 4 Dendrites
13.0M
71.83%
1.57
ResNet-18-prefc-gd + 5 Dendrites
13.3M
71.90%
1.34
ResNet-18-prefc-cascor + 1 Dendrites
12.2M
71.33%
2.98
ResNet-18-prefc-cascor + 2 Dendrites
12.5M
71.73%
2.49
Key Insight: Adding dendrites to ResNet-18 provides 4-7x better accuracy improvement per additional parameter compared to upgrading to ResNet-34. These numbers are acheived without the pre-fc layer where the output FC layer is perforated. That model can be uploaded as well if requested.
Transfer Learning Performance
This model excels at transfer learning tasks:
Model
Parameters
Flowers-102 Accuracy
Oxford Pets Accuracy
Food-101 Accuracy
ResNet-18
11.2M
89.8%
90.8%
81.7%
ResNet-34
21.3M
90.7%
92.6%
83.9%
ResNet-18 Perforated (this model)
12.5M
91.2%
91.8%
82.4%
Result: For Pets and Food-101 this model closes 57% and 30%, respecively, of the accuracy gap with only 13% of the parameter gap between ResNet-18 and ResNet-34. While on the Flowers-102 dataset the perforated resnet-18 outperforms the resnet-34 with over 2x the parameter effeciency.
Full results of all experiments performed can be found here.
Latency
The original model started from the torchvision ResNet rather than the microsoft/resnet. Latency comparisons start there as well to ensure fair results. Test was run processing a single image at a time on a AMD Ryzen Threadripper PRO 9975WX CPU over the full Flowers-102 dataset.
Model
Time Per Image
Throughput
ResNet-18
4.04ms
247.46 FPS
ResNet-34
7.48ms
133.74 FPS
ResNet-18 Perforated
4.37ms
228.63 FPS
As expected, latency is proportinally similar to parameter count between the models.
Usage
Installation
First, install the PerforatedAI library:
pip install perforatedai
Loading the Model
python
1import torch
2import torchvision
3from perforatedai import utils_perforatedai as UPA
4from perforatedai import library_perforatedai as LPA
56# Create base model architecture7base_model = torchvision.models.get_model('resnet18', weights=None, num_classes=1000)89# Convert to perforated architecture10model = LPA.ResNetPAIPreFC(base_model)1112# Load pretrained weights from HuggingFace13model = UPA.from_hf_pretrained(model,'perforated-ai/resnet-18-perforated-cascor')
Using for Transfer Learning
For transfer learning on your own dataset, replace the final FC layer:
python
1import torch.nn as nn
23# Assuming you have num_classes for your task4num_classes =102# Example: Flowers-10256# Replace the final FC layer7model.fc = nn.Linear(model.fc.in_features, num_classes)8
Preprocessing
Use standard preprocessing that you would use with any ResNet-18 such as the following for ImageNet:
Resource-constrained environments where parameter efficiency is critical
Applications requiring ResNet-34 performance with ResNet-18 computational costs
Recommendations
This model is particularly effective when you need strong feature representations without the parameter overhead of larger models. The perforated pre-FC layer has learned rich representations from ImageNet that transfer well to downstream tasks.
Training Details
Training Data
Dataset: ImageNet-1k (1.28M training images, 1000 classes)
Perforation Target: Pre-FC layer (for transfer learning optimization)
Training Procedure
Detailed training code and procedures are available in our GitHub repository:
