You can deploy this model and start earning money today!

yusiwen/
dl-from-scratch

Name: dl-from-scratch
Availability: InStock

Views

No views yet

DL From Scratch

Implement mainstream deep learning models from scratch.

Project Structure

├── main.py
├── pyproject.toml
├── .gitignore
├── README.md
├── ROADMAP.md
├── ml/                    # Classical Machine Learning (pure NumPy)
│   ├── mlp/               # MLP (MNIST, manual backprop)
│   └── basics/            # 12 standalone models (lin/log reg, SVM, K-Means, PCA, RF, GBDT, etc.)
├── cv/                    # Computer Vision
│   ├── simplecnn/         # SimpleCNN (CIFAR-10, Conv×3+Pool×3+FC×2)
│   ├── resnet18/          # ResNet18 (CelebA, 15 attrs, skip connections)
│   ├── resnet34/          # ResNet34 (CelebA, 40 attrs, [3,4,6,3] blocks)
│   ├── resnet50/          # ResNet50 (Bottleneck block 1×1→3×3→1×1)
│   ├── mobilenet/         # MobileNet (depthwise separable conv, CIFAR-10)
│   ├── vit/               # Vision Transformer (patch embed + BERT encoder, CIFAR-10)
│   ├── unet/              # UNet (Oxford-IIIT Pet segmentation)
│   └── yolo/              # YOLO (Pascal VOC object detection)
├── gen/                   # Generative Models
│   ├── dcgan/             # DCGAN (CelebA, transposed conv)
│   ├── vae/               # VAE (reparameterization trick, KL divergence)
│   ├── ddpm/              # DDPM (CIFAR-10, denoising diffusion)
│   └── simclr/            # SimCLR (CIFAR-10, contrastive learning)
├── graph/                 # Graph Neural Networks
│   └── gcn/               # GCN (Cora, spectral graph convolution)
├── rl/                    # Reinforcement Learning
│   └── dqn/               # DQN (CartPole, experience replay)
├── nlp/                   # Natural Language Processing
│   ├── bert/              # BERT (MLM pretrain + classification finetune)
│   ├── gpt/               # GPT (decoder-only, causal attention, KV cache)
│   ├── lstm/              # LSTM (hand-written gates, IMDB sentiment)
│   ├── word2vec/          # Word2Vec (CBOW + Skip-gram, negative sampling)
│   ├── seq2seq/           # Seq2Seq Transformer (EN→DE translation)
│   └── lora/              # LoRA (parameter-efficient GPT fine-tuning)
├── utils/                 # Shared Infrastructure
│   ├── config.py          # YAML config loading/saving
│   ├── seed.py            # Reproducibility seed locking
│   └── device.py          # CUDA → MPS → CPU auto-detection
└── scripts/               # Notebook generation scripts
│   ├── __init__.py
│   ├── config.yaml        # DCGAN hyperparameters
│   ├── model.py           # Generator + Discriminator
│   ├── data.py            # CelebA images (64×64, no labels)
│   ├── train.py           # Adversarial training loop (G/D alternating)
│   └── generate.py        # Generate sample grid from trained model
├── vit/
│   ├── __init__.py
│   ├── config.yaml        # ViT hyperparameters (patch_size, d_model, n_layers, etc.)
│   ├── model.py           # ViT: PatchEmbed → Transformer encoder (reused from BERT) → CLS head
│   ├── data.py            # CIFAR-10 via HF datasets
│   ├── train.py           # Training loop
│   └── eval.py            # Per-class accuracy on test split
├── unet/
│   ├── __init__.py
│   ├── config.yaml        # UNet hyperparameters
│   ├── model.py           # U-Net: encoder–decoder with skip connections
│   ├── data.py            # Oxford-IIIT Pet (image + mask) with augmentation
│   ├── train.py           # Training loop (pixel-wise CrossEntropy)
│   └── eval.py            # IoU and pixel accuracy
├── cnn/
│   ├── __init__.py
│   ├── data.py            # CIFAR-10 via HF datasets (uoft-cs/cifar10)
│   ├── model.py           # Plain CNN (Conv×3 + Pool×3 + FC×2)
│   ├── train.py           # Training script (Adam + CosineAnnealingLR)
│   └── eval.py            # Test evaluation + confusion matrix
├── mlp/
│   ├── __init__.py
│   ├── data.py            # MNIST via HF datasets (ylecun/mnist)
│   ├── model.py           # MLP — pure NumPy (Linear, ReLU, SoftmaxCrossEntropy, SGD)
│   ├── train.py           # Training script
│   └── eval.py            # Test evaluation (per-digit accuracy)
├── utils/
│   ├── __init__.py
│   ├── config.py             # YAML config loading/saving (load_config / save_config)
│   └── seed.py               # set_seed() — lock torch + numpy + random + cudnn
├── nlp/
│   ├── bert/
│   ├── word2vec/
│   ├── lstm/
│   ├── gpt/
│   └── seq2seq/
│       ├── __init__.py
│       ├── tokenizer.py       # Word-level tokenizer (5000 vocab, from text8)
│       ├── model.py           # Decoder-only Transformer (Causal Attention + KV Cache)
│       ├── train.py           # Autoregressive LM on text8
│       └── generate.py        # Text generation (temperature + top-k + [SEP] blocked)
│   └── seq2seq/
│       ├── __init__.py
│       ├── config.yaml        # Transformer hyperparameters
│       ├── model.py           # Encoder (from BERT) + Decoder (cross-attention) → Seq2Seq
│       ├── data.py            # Multi30k EN→DE, word-level tokenizer
│       ├── train.py           # Teacher forcing training
│       └── generate.py        # Greedy decoding translation demo
├── basics/
│   ├── __init__.py
│   ├── logistic_regression.py   # Single Linear layer + Softmax (92.3% on MNIST)
│   ├── linear_regression.py     # California Housing (Normal Equation + GD, R²=0.583)
│   ├── k_means.py               # Unsupervised clustering (pure NumPy)
│   ├── svm.py                   # SVM — GD (primal) + SMO (dual, Linear/RBF kernels)
│   ├── decision_tree.py          # ID3/CART on Iris (ASCII tree, ~93% acc)
│   ├── random_forest.py          # Bagging + random feature subsets, Iris 93.3%
│   ├── gbdt.py                   # Gradient boosting (MSE reg + binary logloss cls)
│   ├── naive_bayes.py            # Gaussian NB on MNIST (generative classifier)
│   ├── pca.py                    # SVD-based dimensionality reduction (MNIST 2D visualisation)
│   ├── knn.py                    # k-Nearest Neighbors (instance-based, MNIST)
│   └── perceptron.py             # Single neuron (Rosenblatt 1958, step activation)
├── .gitattributes                 # LFS: *.zip *.pt
└── uv.lock

Infrastructure

Feature	Description
Config system	Each model directory has a `config.yaml` with its hyperparameters (seed, lr, batch_size, epochs, etc.). Edit the YAML to change training params without touching code.
TensorBoard	Every PyTorch training script logs loss/accuracy per epoch to `runs/{model_name}/`. Run `tensorboard --logdir runs` to visualize all experiments.
Reproducibility	`utils/seed.py` provides `set_seed()` that locks `torch` + `numpy` + `random` + `cudnn`. Called at the start of every train script. Config is saved alongside model weights (`_config.yaml`).

Usage

bash

# View training curves (all models)
tensorboard --logdir runs

# Edit hyperparameters in YAML instead of code
vim cv/resnet18/config.yaml
# then train as usual:
uv run python -m cv.resnet18.train

CV/ResNet18

Item	Value
Model	ResNet18 (11.2M params)
Dataset	CelebA via HF datasets — 1,000 images
Attributes	15 binary (Smiling, Male, Young, Eyeglasses, etc.)
Split	800 train / 200 val
Val Accuracy	91.2%
Training	MPS (Mac M4) + AMP

CV/ResNet34

Item	Value
Model	ResNet34 (~21M params, [3,4,6,3] BasicBlock)
Dataset	CelebA via HF datasets — full 200K
Attributes	All 40 binary attributes
Optimizer	SGD + Momentum (0.9, weight_decay=1e-4)
Training	CosineAnnealingLR + Gradient Accumulation + Early Stopping + Loss Weighting

CV/ResNet50

Item	Value
Model	ResNet50 (~23.6M params, [3,4,6,3] Bottleneck)
Dataset	CelebA via HF datasets — full 200K
Attributes	All 40 binary attributes
Optimizer	SGD + Momentum (0.9, weight_decay=1e-4)
Architecture	Bottleneck block: 1×1 → 3×3 → 1×1 (contrast with BasicBlock's two 3×3)

GEN/VAE

Item	Value
Model	Variational Autoencoder (2.6M params)
Dataset	CelebA via HF datasets — 10K images (64×64)
Architecture	Conv Encoder → μ,logσ² → reparameterize → Deconv Decoder → Sigmoid
Loss	Reconstruction (BCE) + KL divergence
Training	Adam(lr=2e-4), 50 epoch

NLP/Seq2Seq Transformer

Item	Value
Model	Encoder-Decoder Transformer (1M params)
Dataset	Multi30k EN→DE — 29K train / 1K test
Architecture	Encoder (from BERT) + Decoder (causal + cross-attention)
Training	Teacher forcing, weight-tying, Adam(lr=1e-4)

GEN/DDPM

Item	Value
Model	Denoising Diffusion (16.1M params)
Dataset	CIFAR-10 via HF datasets — 50K images (32×32)
Architecture	UNet + timestep embedding + sinusoid positional encoding
Training	Noise prediction (MSE), T=1000, linear β schedule
Sampling	Reverse diffusion (x_T → x_0), 1000 steps

GRAPH/GCN

Item	Value
Model	2-layer Graph Convolutional Network (23K params)
Dataset	Cora via URL — 2708 nodes, 1433 features, 7 classes
Architecture	GraphConv × 2: Â @ H @ W (spectral graph convolution)
Training	Semi-supervised (20 labels/class), CrossEntropyLoss

RL/DQN

Item	Value
Model	Deep Q-Network (17K params)
Environment	CartPole-v1 via Gymnasium — 4-dim state, 2 actions
Architecture	3-layer MLP (4→128→128→2)
Training	Experience replay, target network, ε-greedy decay

GEN/SimCLR

Item	Value
Model	SimCLR (11M params: ResNet18 encoder + MLP projector)
Dataset	CIFAR-10 via HF datasets — self-supervised (no labels)
Architecture	ResNet18 → Projector(512→256→128) → NT-Xent loss
Training	100 epoch, temperature=0.5, dual random augmentation

CV/YOLO

Item	Value
Model	Simplified YOLO (59M params)

NLP/LoRA

Item	Value
Model	Low-Rank Adaptation on GPT (32K trainable / 5.7M frozen)
Dataset	text8 via HF datasets — 5K chunks
Architecture	LoRALayer: frozen Linear + low-rank B×A
Key concept	Parameter-efficient fine-tuning, 0.58% trainable params
Comparison	Full fine-tune: 5.7M vs LoRA r=8: 32K

CV/MobileNet

Item	Value
Model	MobileNetV1 (135K params, width=1.0)
Dataset	CIFAR-10 via HF datasets — 50K train / 10K test
Architecture	DepthwiseSeparableConv (depthwise 3×3 + pointwise 1×1)
Key concept	Depthwise separable convolution, ~8.4× fewer ops than standard conv
Comparison	SimpleCNN 620K params → MobileNet 135K (4.6× smaller)
Dataset	Pascal VOC via HF datasets — 20 classes
Architecture	CNN backbone → FC detection head → 7×7×30 output
Training	YOLO loss (coord + obj + noobj + class), NMS at inference

DCGAN

Item	Value
Model	Generator (3.5M params) + Discriminator (2.8M params)
Dataset	CelebA via HF datasets — 10K images (64×64)
Architecture	Transposed conv G / Conv D, BN, LeakyReLU
Optimizer	Adam(lr=2e-4, β₁=0.5) — separate for G and D
Training	BCELoss, label smoothing, fixed noise grid for monitoring

CV/ViT

Item	Value
Model	Vision Transformer (807K params, 4 layers, 4 heads, 128-dim)
Dataset	CIFAR-10 via HF datasets — 50K train / 10K test
Architecture	PatchEmbed(4×4) → [CLS] → Transformer Encoder (from BERT) → CLS head
Key concept	Self-attention for vision, no convolutions, patch embeddings

CV/UNet

Item	Value
Model	U-Net (31M params, 5 encoder/decoder stages)
Dataset	Oxford-IIIT Pet via HF datasets — image + segmentation mask
Architecture	Encoder: Conv+MaxPool × 4, Decoder: UpConv+skip × 4, output: pixel-wise logits
Loss	CrossEntropy (ignore_index=0 for unlabeled)
Metrics	Pixel accuracy, mean IoU

CV/SimpleCNN

Item	Value
Model	SimpleCNN (620K params)
Dataset	CIFAR-10 via HF datasets — 50K images
Classes	10 (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck)
Test Accuracy	82.4% (30 epochs)
Training	Adam + CosineAnnealingLR

ML/MLP

Item	Value
Model	MLP (235K params, pure NumPy)
Dataset	MNIST via HF datasets — 60K images
Classes	10 digits (0-9)
Test Accuracy	97.9% (20 epochs)
Framework	NumPy only (hand-written backward pass)

BERT

Item	Value
Model	BERT mini (834K params, 4 layers, 4 heads, 128-dim)
Pre-training	MLM on text8 (90M chars, HuggingFace)
Fine-tuning	Sentiment classification on IMDB (HuggingFace)
Test Accuracy	~50% (character-level; word-level would be higher with subword tokenization)
Core components	Self-Attention (semantic aggregation) + MLM (entropy increase noise reduction)

Word2Vec

Item	Value
Model	Word2Vec (50-dim embeddings, 97K vocab)
Architectures	CBOW + Skip-gram with Negative Sampling
Dataset	text8 via HF datasets (~90M chars)
Training	Adam, 5 epochs, k=5 negative samples
Evaluation	Cosine similarity search in embedding space
Key concept	Static word embeddings from distributional semantics

LSTM

Item	Value
Model	LSTM (145K params, hand-written gates)
Dataset	IMDB via HuggingFace (9K train / 1K test)
Architecture	Embedding(128) → LSTM(128→128) → FC(128→2)
Test Accuracy	~50-60% (character-level, harder than word-level)
Key concepts	Input/forget/output gates, cell state, gradient flow through gating

GPT

Item	Value
Model	Decoder-only Transformer (5.7M params, word-level)
Dataset	text8 via HuggingFace (15M words, 20K chunks)
Training	Autoregressive (predict next token), PPL 4.63
Generation	Temperature + top-k sampling with KV Cache, [SEP] blocked
Key concepts	Causal Self-Attention, KV Cache, autoregressive generation, word-level tokenization

Basics

Algorithm	File	Datasets	Metric
Logistic Regression	`ml/basics/logistic_regression.py`	MNIST	92.3% test accuracy
Linear Regression	`ml/basics/linear_regression.py`	California Housing	R²=0.583
K-Means	`ml/basics/k_means.py`	MNIST	57.8% cluster purity
SVM (GD + SMO)	`ml/basics/svm.py`	MNIST 3v5	93.3% (RBF kernel)
Decision Tree	`ml/basics/decision_tree.py`	Iris	93.3% test acc
Random Forest	`ml/basics/random_forest.py`	Iris	93.3% test acc (50 trees)
GBDT	`ml/basics/gbdt.py`	sin(x) / synthetic 2D	MSE 0.21 / 100% bin cls
Naive Bayes	`ml/basics/naive_bayes.py`	MNIST	53.0% test acc
PCA	`ml/basics/pca.py`	MNIST	17.3% variance in 2 components
k-NN	`ml/basics/knn.py`	MNIST	~87% (k=5, 2000 train)
Perceptron	`ml/basics/perceptron.py`	MNIST 0v1	100% (linearly separable)

SVM implementations

Method	Type	Kernel	Notes
`SVM_GD`	Primal GD	Linear only	Fast, robust, ~80 lines
`SVM_SMO`	Dual SMO	Linear + RBF	Platt SMO, ~150 lines, supports kernel trick

See resnet18/README.md for details.

Core Concepts

Every model in this project was written from scratch to teach a specific set of ML/DL concepts. The table below maps each model to the key ideas it demonstrates.

Module	Model	Key concepts
`basics/`	Logistic Regression	Linear decision boundary, Softmax, Cross-Entropy, closed-form vs gradient descent
`basics/`	Linear Regression	Normal Equation, MSE, R² score, feature standardisation
`basics/`	K-Means	Unsupervised learning, Euclidean distance, iterative centroid refinement, cluster purity
`basics/`	SVM (GD)	Hinge loss, max-margin classification, L2 regularisation, primal gradient descent
`basics/`	SVM (SMO)	Dual formulation, Lagrange multipliers, KKT conditions, kernel trick (RBF)
`basics/`	Decision Tree	Entropy, Information Gain, recursive partitioning, interpretable ASCII tree
`basics/`	Random Forest	Bagging, bootstrap sampling, feature randomness, ensemble diversity → variance reduction
`basics/`	GBDT	Gradient boosting, stage-wise additive model, pseudo-residuals, learning rate shrinkage
`basics/`	Naive Bayes	Bayes' theorem, generative vs discriminative models, Gaussian likelihood, log-space prediction
`basics/`	PCA	Singular Value Decomposition (SVD), eigenvalue, dimensionality reduction, variance explained
`basics/`	k-NN	Instance-based learning, distance metrics, curse of dimensionality, bias-variance tradeoff
`basics/`	Perceptron	Single neuron, step activation, online learning, Perceptron Convergence Theorem
`mlp/`	MLP (NumPy)	Manual backpropagation, chain rule, gradient descent without autograd, softmax cross-entropy
`cv/simplecnn/`	SimpleCNN	Convolution, max-pooling, BatchNorm, Dropout, CosineAnnealing LR schedule
`cv/resnet18/`	ResNet18	Residual connections (skip connections), BatchNorm in deep networks, bottleneck design, AMP
`cv/resnet34/`	ResNet34	SGD+Momentum, CosineAnnealingLR, gradient accumulation, early stopping, ROC AUC, F1
`cv/resnet50/`	ResNet50	Bottleneck block (1×1→3×3→1×1), deeper residual networks
`gen/vae/`	VAE	Reparameterization trick, KL divergence, latent space interpolation
`nlp/seq2seq/`	Seq2Seq Transformer	Encoder-decoder, cross-attention, teacher forcing, weight-tying
`gen/ddpm/`	DDPM	Denoising Diffusion, UNet + timestep embedding, noise prediction
`gen/dcgan/`	DCGAN	Transposed convolution, adversarial training, generator/discriminator dynamics
`cv/vit/`	Vision Transformer (ViT)	Patch embedding, self-attention for vision, Transformer without convolutions
`cv/unet/`	U-Net	Encoder-decoder, skip connections, pixel-wise classification, IoU metric
`nlp/bert/`	BERT mini	Self-Attention (semantic aggregation), Masked Language Model (entropy increase + denoising), LayerNorm, positional encoding
`nlp/word2vec/`	Word2Vec	Embedding lookup tables, Negative Sampling, CBOW vs Skip-gram, subsampling frequent words, cosine similarity
`nlp/lstm/`	LSTM	Input/forget/output gates, cell state, gradient flow through gating, sequential processing vs parallel attention
`graph/gcn/`	GCN	Graph convolution, message passing, semi-supervised node classification
`rl/dqn/`	DQN	Q-Learning, experience replay, target network, ε-greedy
`gen/simclr/`	SimCLR	Contrastive learning, NT-Xent loss, data augmentation
`nlp/lora/`	LoRA	Low-rank adaptation, parameter-efficient fine-tuning, GPT adapter
`cv/mobilenet/`	MobileNet	Depthwise separable convolution, efficient CNN, width multiplier
`cv/yolo/`	YOLO	Single-stage object detection, grid-based regression, NMS
`nlp/gpt/`	GPT	Causal Self-Attention, KV Cache, autoregressive generation, word-level tokenizer, temperature + top-k sampling, bad-token blocking

Setup & Run

uv sync