Views
No views yet
| Architecture | Mamba-2 SSM (state-space model) |
| Parameters | ~300M |
| Optimizer | AdamW |
| Data perturbation | standard (unperturbed) text |
| Training data | FineWeb-Edu sample-10BT (~9.4B tokens) |
| Context length | 1024 |
| Tokenizer | Llama 3 (128K vocab) |
| Batch size | 512 sequences |
tokens-200M, tokens-400M, ..., tokens-9.6B.1from transformers import AutoModelForCausalLM
2
3# Load final checkpoint
4model = AutoModelForCausalLM.from_pretrained("deqing/convergent-mamba2-300M-adamw-original")
5
6# Load intermediate checkpoint (e.g., at 1B tokens)
7model = AutoModelForCausalLM.from_pretrained("deqing/convergent-mamba2-300M-adamw-original", revision="tokens-1B")