Model Card for Llama-3-8B-Instruct-SkillMix

Training Details

• LR: 2e-5
  • Linear Warmup Ratio: 0.03
  • Decay: Cosine Decay to 0
• Batch Size: 128
• epoch: 7 / 15
• Optimizer: AdamW
• Sequence Length: 1024

Evaluation Details

AlpacaEval

• model_kwargs:
  • torch_dtype: 'bfloat16'
  • max_new_tokens: 2048
• temperature: 0.9
• top_p: 1.0
• do_sample: True
• stop_token_ids:
  • 128001
  • 128009

MTBench

• model_kwargs:
  • torch_dtype: 'bfloat16'
  • max_new_tokens: 1024
• temperature: 0.7
• stop_token_ids:
  • 128001
  • 128009

WildBench

• model_kwargs:
  • torch_dtype: 'bfloat16'
  • max_new_tokens: 4096
• temperature: 0.9
• top_p: 1.0
• do_sample: True
• stop_token_ids:
  • 128001
  • 128009

Citation

@misc{kaur2024instructskillmixpowerfulpipelinellm,
      title={Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning}, 
      author={Simran Kaur and Simon Park and Anirudh Goyal and Sanjeev Arora},
      year={2024},
      eprint={2408.14774},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2408.14774}, 
}

Contact