Views
No views yet
| Base model | meta-llama/Llama-3.2-3B |
| Data | Nemotron-CC-Math-4plus (52B tokens) |
| Epochs | 1 |
| Sequence length | 2,048 |
| Effective batch size | 128 sequences (262K tokens) |
| Learning rate | 2e-5, cosine decay, 5% warmup |
| Optimizer | AdamW, weight decay 0.01, grad clip 1.0 |
| Precision | BF16 + Flash Attention 2 |
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model = AutoModelForCausalLM.from_pretrained("pavelslab-nyu/Llama-3.2-3B-CPT-Math")
4tokenizer = AutoTokenizer.from_pretrained("pavelslab-nyu/Llama-3.2-3B-CPT-Math")1@article{rahman2026when,
2 title = {When Can LLMs Learn to Reason with Weak Supervision?},
3 author = {Rahman, Salman and Shen, Jingyan and Mordvina, Anna and
4 Palangi, Hamid and Gabriel, Saadia and Izmailov, Pavel},
5 journal = {Preprint},
6 year = {2026}
7}