Llama3.1-SuperDeepFuse-CrashCourse12K

Model Details

• Base Model: Llama3.1-SuperDeepFuse (8B parameters)
• Fine-tuning Dataset: 12 000 samples from agentlans/crash-course (containing samples from 10 high-quality instruct datasets)
• Model Type: Instruction-tuned language model
• Language(s): Multilingual
• License: Follows standard Llama 3.1 usage terms

Training Procedure

Fine-tuning

• Method: LoRA (Low-Rank Adaptation)
• Optimizer: AdamW
• Learning Rate: 5e-5
• Batch Size: 2 per device
• Gradient Accumulation Steps: 8
• Training Epochs: 1
• Max Sequence Length: 2048
• LoRA Configuration:
  • Rank: 8
  • Alpha: 16
  • Dropout: 0.5
  • Target: all layers
• Quantization: 4-bit (bitsandbytes)
• Precision: BF16
• Other Techniques: NEFTune (noise alpha: 5), RS-LoRA

Performance and Limitations

• Enhanced multi-task reasoning
• Improved performance in mathematics and coding tasks
• Better instruction-following abilities

• Performance may be limited compared to larger model variants
• Can produce misleading or incorrect outputs
• Outputs should be independently verified for critical applications

Additional Information

• For the original model, see agentlans/Llama3.1-SuperDeepFuse
• For the base Llama 3.1 model, including training data and model architecture, refer to the original Llama 3.1 model card.

Open LLM Leaderboard Evaluation Results