You can deploy this model and start earning money today!

Prisma-Multimodal/
sparse-autoencoder-clip-b-32-sae-vanilla-x64-layer-2-hook_resid_post-l1-5e-05

Name: sparse-autoencoder-clip-b-32-sae-vanilla-x64-layer-2-hook_resid_post-l1-5e-05
Availability: InStock

mechanistic interpretability

feature-extraction

apache-2.0

Views

No views yet

CLIP-B-32 Sparse Autoencoder x64 vanilla - L1:5e-05

Training Details

Base Model: CLIP-ViT-B-32 (LAION DataComp.XL-s13B-b90K)
Layer: 2
Component: hook_resid_post

Model Architecture

Input Dimension: 768
SAE Dimension: 49,152
Expansion Factor: x64 (vanilla architecture)
Activation Function: ReLU
Initialization: encoder_transpose_decoder
Context Size: 50 tokens

Performance Metrics

L1 Coefficient: 5e-05
L0 Sparsity: 716.8327
Explained Variance: 0.9060 (90.60%)

Training Configuration

Learning Rate: 0.0004
LR Scheduler: Cosine Annealing with Warmup (200 steps)
Epochs: 10
Gradient Clipping: 1.0
Device: NVIDIA Quadro RTX 8000

Experiment Tracking:

Weights & Biases Run ID: 9ve8o3it
Full experiment details: https://wandb.ai/perceptual-alignment/clip/runs/9ve8o3it/overview
Git Commit: e22dd02726b74a054a779a4805b96059d83244aa

Citation

bibtex

@misc{2024josephsparseautoencoders,
    title={Sparse Autoencoders for CLIP-ViT-B-32},
    author={Joseph, Sonia},
    year={2024},
    publisher={Prisma-Multimodal},
    url={https://huggingface.co/Prisma-Multimodal},
    note={Layer 2, hook_resid_post, Run ID: 9ve8o3it}
}

Prisma-Multimodal/sparse-autoencoder-clip-b-32-sae-vanilla-x64-layer-2-hook_resid_post-l1-5e-05