outputs/qwen2.5-0.5b-ft

Model description

• Enhanced knowledge and capabilities in coding and mathematics
• Improved instruction following and long text generation (over 8K tokens)
• Better understanding of structured data and generation of structured outputs (especially JSON)
• Increased resilience to diverse system prompts
• Long-context support up to 128K tokens with the ability to generate up to 8K tokens
• Multilingual support for over 29 languages

• Type: Causal Language Model
• Training Stage: Pretraining
• Architecture: Transformers with RoPE, SwiGLU, RMSNorm, Attention QKV bias, and tied word embeddings
• Number of Parameters: 0.49B (0.36B non-embedding)
• Number of Layers: 24
• Number of Attention Heads (GQA): 14 for Q and 2 for KV
• Context Length: Full 32,768 tokens

Intended uses & limitations

• As a base language model, it is not recommended for direct use in conversations without further fine-tuning or post-training techniques like SFT or RLHF.
• The model's performance may vary across different languages and domains.
• Users should be aware of potential biases present in the training data.

Training and evaluation data

• A diverse range of open-source datasets
• Custom-created synthetic datasets
• 1 million primarily synthetically generated instruction and chat samples
• High-quality, curated content that has contributed to the advancements in SOTA LLMs

Training procedure

Training hyperparameters

• learning_rate: 1e-05
• train_batch_size: 5
• eval_batch_size: 5
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 40
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• num_epochs: 3
• weight_decay: 0.01

• Gradient Checkpointing: Enabled
• Mixed Precision: BF16 (auto)
• Sequence Length: 4096
• Sample Packing: Enabled
• Pad to Sequence Length: Enabled

Framework versions

• Transformers 4.45.0.dev0
• Pytorch 2.3.1+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1

Additional Information