Beta
Explore
Marketplace
Neural Labs
Chat
Wallet
Docs
sae-clip-b32-x64-layer10-resid-post-cls-lr1e-2 – AI Model by orrav | AlphaNeural AI
You can deploy this model and start earning money today!
orrav
/
sae-clip-b32-x64-layer10-resid-post-cls-lr1e-2
like
0
torch
clip
vision
transformers
interpretability
sparse autoencoder
sae
mechanistic interpretability
feature-extraction
en
us
Views
No views yet
Model card
Files and Versions
Community
API
Deploy
CLIP-B-32 Sparse Autoencoder x64 vanilla - L1:1e-05
Training Details
Base Model: CLIP-ViT-B-32 (LAION DataComp.XL-s13B-b90K)
Layer: 10
Component: hook_resid_post
Model Architecture
Input Dimension: 768
SAE Dimension: 49,152
Expansion Factor: x64 (vanilla architecture)
Activation Function: ReLU
Initialization: encoder_transpose_decoder
CLS_only: true
Performance Metrics
L1 Coefficient: 1e-05
L0 Sparsity: 655.9
Explained Variance: 89.42%
Training Configuration
Learning Rate: 0.01
LR Scheduler: Cosine Annealing with Warmup (200 steps)
Epochs: 10
Gradient Clipping: 1.0
