FinLLama-7B SAE Models: Layers 4, 10, 16, 22, 28

🚀 Model Overview

• Base Model: cxllin/Llama2-7b-Finance (Financial fine-tuned from meta-llama/Llama-2-7b-hf)
• Training Dataset: WikiText-103
• Architecture: Sparse Autoencoder (SAE)
• Layers Trained: 4, 10, 16, 22, 28 (5 layers at regular intervals)
• Configuration: k=32, 400 latents, Context=1024, LR=0.001, Batch=8
• Model Type: Financial domain fine-tuned model

📊 Model Performance Summary

🔍 Key Findings

Best Performing Layer

• Layer 16: Only layer meeting loss recovery threshold (62.89%)
• Most Balanced: 3/4 healthy metrics, best overall performance

Training Success Rate

• Fully Healthy: 0/5 layers (0%)
• Partially Healthy: 1/5 layers (20%)
• Unhealthy: 4/5 layers (80%)
• Overall: Challenging training compared to base model

📁 Repository Structure

llama2_7b_finance_layers4_10_16_22_28_k32_latents400_wikitext103_torchrun/
├── config.json                    # Model configuration
├── state.pt                      # Training state
├── optimizer_0.pt               # Optimizer state
├── lr_scheduler_0.pt            # Learning rate scheduler
├── rank_0_state.pt              # Distributed training state
├── layers.4/                    # Layer 4 SAE model
├── layers.10/                   # Layer 10 SAE model
├── layers.16/                   # Layer 16 SAE model
├── layers.22/                   # Layer 22 SAE model
└── layers.28/                   # Layer 28 SAE model

🚀 Usage

Loading Individual Layer Models

Model Configuration

• k: 32 (sparsity parameter)
• Latents: 400 (bottleneck dimension)
• Context Length: 1024 tokens
• Learning Rate: 0.001
• Batch Size: 8

📈 Training Details

Dataset

• Primary: WikiText-103 (English language modeling)
• Base Model: Financial domain fine-tuned from Llama-2-7B

Hardware & Training

• Framework: PyTorch with torchrun
• Configuration: Same as base Llama-2-7B for comparison

Hyperparameters

• Optimizer: AdamW
• Scheduler: Linear learning rate decay
• Gradient Accumulation: Enabled for stability
• Mixed Precision: FP16 for efficiency

🔬 Model Analysis

Sparsity Analysis

• L0 Sparsity Range: 42.12 - 55.11
• Optimal Range: 20-200 (SAEBench standards)
• Consistency: Excellent sparsity control across all layers

Feature Health

• Dead Features: 65.32-89.12% (above 50% threshold)
• Absorption Values: 0.265-0.312 (within 0.25-0.35 range)
• Feature Utilization: Challenging due to high dead features

🆚 Comparison with Base Llama-2-7B

🎯 Key Insights

Strengths

1. Consistent Sparsity Control: L0 values consistently within SAEBench sweet spot
2. Stable Architecture: Minimal variation in L0 sparsity across layer depths
3. Layer 16 Viability: Only layer suitable for SAE applications
4. Predictable Behavior: Clear pattern of performance across layers

Challenges

1. High Dead Features: Critical issue affecting model utility
2. Lower Loss Recovery: Below SAEBench thresholds for most layers
3. Training Difficulty: Financial fine-tuning impacts SAE trainability

📚 Research Implications

Domain Fine-tuning Effects

• Financial specialization appears to reduce SAE trainability
• Feature sparsity is maintained but feature utilization suffers
• Layer 16 shows potential for financial domain SAE applications

Recommendations

1. Use Layer 16 for financial SAE applications
2. Optimize hyperparameters for fine-tuned models
3. Consider domain-specific training strategies
4. Base model superior for general SAE applications

📚 References

• SAE Training: Based on SAEBench methodology
• Base Model: cxllin/Llama2-7b-Finance (financial fine-tuned)
• Dataset: WikiText-103 for language modeling
• Comparison: Analysis vs. base Llama-2-7B SAE models

🤝 Contributing

📄 License