Views
No views yet
fp16=False and bf16=True in the HF trainer. Gathering more intel on this but if you have any thoughts about this issue or performance, please let us know!
| Metric | Value |
|---|---|
| MMLU (5-shot) | - |
| ARC (25-shot) | - |
| HellaSwag (10-shot) | - |
| TruthfulQA (0-shot) | - |
| Avg. | - |
### Instruction:
<prompt> (without the <>)
### Response:garage-bAInd/Platypus2-7B trained using STEM and logic based dataset garage-bAInd/Open-Platypus.garage-bAInd/Platypus2-7B was instruction fine-tuned using LoRA on 1 A100 80GB. For training details and inference instructions please see the Platypus2 GitHub repo.# clone repository
git clone https://github.com/EleutherAI/lm-evaluation-harness.git
# check out the correct commit
git checkout b281b0921b636bc36ad05c0b0b0763bd6dd43463
# change to repo directory
cd lm-evaluation-harness
# install
pip install -e .python main.py --model hf-causal-experimental --model_args pretrained=garage-bAInd/Platypus2-7B,use_accelerate=True,dtype="bfloat16" --tasks arc_challenge --batch_size 2 --no_cache --write_out --output_path results/Platypus2-7B/arc_challenge_25shot.json --device cuda --num_fewshot 25python main.py --model hf-causal-experimental --model_args pretrained=garage-bAInd/Platypus2-7B,use_accelerate=True,dtype="bfloat16" --tasks hellaswag --batch_size 2 --no_cache --write_out --output_path results/Platypus2-7B/hellaswag_10shot.json --device cuda --num_fewshot 10python main.py --model hf-causal-experimental --model_args pretrained=garage-bAInd/Platypus2-7B,use_accelerate=True,dtype="bfloat16" --tasks hendrycksTest-* --batch_size 2 --no_cache --write_out --output_path results/Platypus2-7B/mmlu_5shot.json --device cuda --num_fewshot 5python main.py --model hf-causal-experimental --model_args pretrained=garage-bAInd/Platypus2-7B,use_accelerate=True,dtype="bfloat16" --tasks truthfulqa_mc --batch_size 2 --no_cache --write_out --output_path results/Platypus2-7B/truthfulqa_0shot.json --device cuda1@article{platypus2023,
2 title={Platypus: Quick, Cheap, and Powerful Refinement of LLMs},
3 author={Ariel N. Lee and Cole J. Hunter and Nataniel Ruiz},
4 booktitle={arXiv preprint arxiv:2308.07317},
5 year={2023}
6}1@misc{touvron2023llama,
2 title={Llama 2: Open Foundation and Fine-Tuned Chat Models},
3 author={Hugo Touvron and Louis Martin and Kevin Stone and Peter Albert and Amjad Almahairi and Yasmine Babaei and Nikolay Bashlykov year={2023},
4 eprint={2307.09288},
5 archivePrefix={arXiv},
6}1@inproceedings{
2 hu2022lora,
3 title={Lo{RA}: Low-Rank Adaptation of Large Language Models},
4 author={Edward J Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen},
5 booktitle={International Conference on Learning Representations},
6 year={2022},
7 url={https://openreview.net/forum?id=nZeVKeeFYf9}
8}