open-llama-3b-openthought-mid-lora

Training

• Base: open_llama_3b_v2 (4-bit NF4)
• Method: QLoRA (r=32, alpha=32)
• Data: open-thoughts/OpenThoughts-114k (filtered <= 2024 tokens, 10,582 samples)
• Epochs: 3
• Batch size: 128
• LR: 2e-4 (cosine, warmup 24 steps)
• Loss: full sequence (pad masked)
• Packing: greedy bin packing sorted by length
• Context: 2024 (native, no RoPE scaling)
• Special tokens: Qwen3.5 chat template
• modules_to_save: embed_tokens, lm_head
• Final loss: 0.71

Format

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
{question}<|im_end|>
<|im_start|>assistant
<think>
{reasoning}
</think>

{answer}<|im_end|>

Merged model