Model Card for SwarmFormer-Base

Model Details

Model Description

• Token embedding layer with heavy dropout (0.4)
• Multiple SwarmFormer layers
• Mean pooling layer
• Final classification layer
• Comprehensive dropout throughout (0.3-0.4)
• Developed by: Jordan Legg, Mikus Sturmanis, Takara.ai
• Funded by: Takara.ai
• Shared by: Takara.ai
• Model type: Hierarchical transformer
• Language(s): English
• License: Not specified
• Finetuned from model: Trained from scratch

Model Sources

• Repository: https://github.com/takara-ai/SwarmFormer
• Paper: "SwarmFormer: Local-Global Hierarchical Attention via Swarmed Token Representations"
• Demo: Not available

Uses

Direct Use

• Text classification
• Sentiment analysis
• Document processing

Downstream Use

• Feature extraction for NLP tasks
• Transfer learning
• Building block for larger systems

Out-of-Scope Use

• Text generation
• Machine translation
• Tasks requiring >768 tokens
• Real-time processing without adequate hardware

Bias, Risks, and Limitations

• Fixed cluster size (4 tokens)
• Maximum sequence length: 768 tokens
• Potential information loss in clustering
• Limited evaluation (English text classification only)

Training Details

Training Data

• Dataset: IMDB Movie Review (50k samples)
• Augmentation techniques:
  • Sentence-level shuffling
  • Controlled synonym replacement
  • Hierarchical sample creation

Training Procedure

Model Architecture Details

1. Token Embedding Layer:
   python

   1- Embedding layer (vocab_size → d_model) 2- Dropout rate: 0.4
2. Local Swarm Aggregator:
   python

   1- Input processing dropout: 0.3 2- Local aggregation MLP: 3 - Linear(d_model → d_model) 4 - GELU activation 5 - Dropout(0.3) 6 - Linear(d_model → d_model) 7- Gate network: 8 - Linear(2*d_model → d_model) 9 - GELU activation 10 - Linear(d_model → d_model) 11 - Sigmoid activation 12- Output dropout: 0.3
3. Clustering Mechanism:
   • Groups tokens into fixed-size clusters (size=4)
   • Computes mean representation per cluster
4. Global Cluster Attention:
   python

   1- Query/Key/Value projections: Linear(d_model → d_model) 2- Scaled dot-product attention 3- Attention dropout: 0.3 4- Output dropout: 0.3
5. Broadcast Updater:
   python

   1- Linear projection: d_model → d_model 2- Dropout: 0.1 3- Gate network: 4 - Linear(2*d_model → d_model) 5 - GELU activation 6 - Linear(d_model → d_model) 7 - Sigmoid activation

Training Hyperparameters

• Embedding dimension: 192
• Number of layers: 2
• Local update steps (T_local): 3
• Cluster size: 4
• Batch size: 48
• Learning rate: 4.74 × 10⁻⁴
• Weight decay: 0.0381
• Dropout rates:
  • Embedding: 0.4
  • Local aggregation: 0.3
  • Attention: 0.3
  • Final: 0.4

Evaluation

Testing Data, Factors & Metrics

• IMDB test split (25k samples)
• Full FP32 inference
• Batch size: 256

Results

• Accuracy: 89.03%
• Precision: 87.22%
• Recall: 91.46%
• F1: 89.29%
• Mean batch latency: 4.83ms
• Peak memory: 9.13GB

Technical Specifications

Model Architecture and Objective

1. Input → Token Embedding (with dropout)
2. For each layer:
   • Multiple iterations of Local Swarm Updates
   • Cluster Formation
   • Global Attention between clusters
   • Broadcast updates back to tokens
3. Mean pooling across sequence
4. Final dropout and classification

Compute Infrastructure

• GPU: NVIDIA RTX 2080 Ti or equivalent
• VRAM: 10GB+ recommended
• Framework: PyTorch

Software Requirements

Citation

Model Card Authors

Model Card Contact