Views
No views yet
1Model Dimensions:
2 - d_model: 768
3 - n_heads: 12 (for compatibility, not used in Mamba)
4 - d_ff: 3072
5 - H_layers: 12 (high-level hierarchy)
6 - L_layers: 12 (low-level processing)
7
8Mamba2 Settings:
9 - d_state: 128
10 - expand: 2
11 - headdim: 64
12 - d_conv: 4
13 - ngroups: 1
14
15Training:
16 - Max halt steps: 8
17 - Block size: 1024
18 - Batch size: 32 (effective)
19 - Learning rate: 0.0002 → 1e-06
20 - Weight decay: 0.1t5-small (T5 SentencePiece)8.12163366.371from transformers import T5Tokenizer
2from hrm_text1_modeling import HRMText1
3
4tokenizer = T5Tokenizer.from_pretrained("t5-small")
5model = HRMText1.from_pretrained("Viharikvs/CMBA-768M-FineWeb")
6
7# Generate text
8input_ids = tokenizer("Once upon a time", return_tensors="pt").input_ids
9outputs = model.generate(input_ids, max_length=100)
10print(tokenizer.decode(outputs[0]))1@misc{cmba-768m-fineweb,
2 author = {Vihari},
3 title = {CMBA-768M-FineWeb: Hierarchical Mamba-based Language Model},
4 year = {2025},
5 publisher = {HuggingFace},
6 url = {https://huggingface.co/Viharikvs/CMBA-768M-FineWeb}
7}