Gated PixelCNN from Scratch — CIFAR-10
A complete from-scratch implementation of the Gated PixelCNN autoregressive generative model, trained on CIFAR-10.
Built using Huggingface's Ml-intern.
🚀 Quick Start
- Open
Gated_PixelCNN_CIFAR10.ipynb in Google Colab
- Select Runtime → Change runtime type → T4 GPU
- Run all cells — training takes ~2 hours
Architecture
Implemented entirely from scratch using only torch.nn primitives:
- Masked Convolutions with correct Mask A/B and RGB sub-pixel channel ordering
- Vertical + Horizontal Stack architecture eliminating the blind spot
- Gated Activations:
y = tanh(W_f * x) ⊙ σ(W_g * x)
- 256-way categorical cross-entropy loss per pixel per channel
| Component | Specification |
|---|
| Model | Gated PixelCNN |
| Layers | 15 gated layers |
| Filters | 128 per stack |
| Kernels | 7×7 input, 3×3 body |
| Parameters | ~5.4M |
| Loss | Cross-Entropy (256 classes) |
| Metric | BPD + FID |
Evaluation
| Metric | Our Model (50 epochs) | Paper [2] (converged) |
|---|
| BPD (test) | ~3.5-4.0 | 3.03 |
| FID | Reported in notebook | — |
T4 GPU Optimization
- Total VRAM usage: ~2 GB (model + activations)
- Batch size: 32
- Training time: ~2 hours for 50 epochs
- fp32 (model is small enough; fp16 gains minimal)
References
- van den Oord et al., "Pixel Recurrent Neural Networks," ICML 2016. arXiv:1601.06759
- van den Oord et al., "Conditional Image Generation with PixelCNN Decoders," NeurIPS 2016. arXiv:1606.05328
- Salimans et al., "PixelCNN++," ICLR 2017. arXiv:1701.05517
