Qwen3-1.7B Macedonian Pretraining

Training details

• Model: Qwen3-1.7B
• Training objective: causal language modeling
• Data: Macedonian corpus
• Epochs: 3
• Checkpoint: checkpoint-128244 / final checkpoint

Limitations

Hardware and training setup

• Base model: Qwen/Qwen3-1.7B-Base
• Pretraining type: continued pretraining on Macedonian text
• Dataset: LVSTCK/macedonian-corpus-cleaned-dedup
• Preprocessing: tokenized with block_size=2048, 10,000 examples held out for test/validation
• Precision: bfloat16
• Attention: flash_attention_2
• Optimizer: adamw_torch_fused
• Sequence length: 2048
• Epochs: 3
• Train batch size: 4 per device
• Eval batch size: 2 per device
• Gradient accumulation: 8
• Gradient checkpointing: disabled
• Warmup steps: 1000
• Max gradient norm: 1.0
• Seed: 2504