Beta
Explore
Marketplace
Neural Labs
Chat
Wallet
Docs
sae-clip-b32-x16-layer11-resid-post-cls – AI Model by orrav | AlphaNeural AI
You can deploy this model and start earning money today!
orrav
/
sae-clip-b32-x16-layer11-resid-post-cls
like
0
torch
clip
vision
transformers
interpretability
sparse autoencoder
sae
mechanistic interpretability
feature-extraction
en
us
Views
No views yet
Model card
Files and Versions
Community
API
Deploy
CLIP-B-32 Sparse Autoencoder x64 vanilla - L1:1e-05
Training Details
Base Model: CLIP-ViT-B-32 (LAION DataComp.XL-s13B-b90K)
Layer: 11
Component: hook_resid_post
Model Architecture
Input Dimension: 768
SAE Dimension: 49,152
Expansion Factor: x64 (vanilla architecture)
Activation Function: ReLU
Initialization: encoder_transpose_decoder
Context Size: 50 tokens
CLS_only: true
Performance Metrics
L1 Coefficient: 1e-05
L0 Sparsity: 10.8
Explained Variance: 0.077%
Training Configuration
Learning Rate: 0.0004
LR Scheduler: Cosine Annealing with Warmup (200 steps)
Epochs: 1
Gradient Clipping: 1.0
