• Model creator: https://huggingface.co/abhinand/
• Original model: https://huggingface.co/abhinand/TinyLlama-1.1B-OpenHermes-2.5-Chat-v0.1-sft/

Original model description:

• en license: apache-2.0 datasets:
• teknium/OpenHermes-2.5
• abhinand/ultrachat_200k_sharegpt model-index:
• name: TinyLlama-1.1B-OpenHermes-2.5-Chat-v0.1-sft results:
  • task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics:
    • type: acc_norm value: 33.79 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=abhinand/TinyLlama-1.1B-OpenHermes-2.5-Chat-v0.1-sft name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics:
    • type: acc_norm value: 58.72 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=abhinand/TinyLlama-1.1B-OpenHermes-2.5-Chat-v0.1-sft name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics:
    • type: acc value: 24.52 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=abhinand/TinyLlama-1.1B-OpenHermes-2.5-Chat-v0.1-sft name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics:
    • type: mc2 value: 36.22 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=abhinand/TinyLlama-1.1B-OpenHermes-2.5-Chat-v0.1-sft name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics:
    • type: acc value: 60.93 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=abhinand/TinyLlama-1.1B-OpenHermes-2.5-Chat-v0.1-sft name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics:
    • type: acc value: 5.38 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=abhinand/TinyLlama-1.1B-OpenHermes-2.5-Chat-v0.1-sft name: Open LLM Leaderboard

---

TinyLLaMA OpenHermes2.5 [Work in Progress]

See axolotl config

axolotl version: 0.4.0

yaml

1base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T 2model_type: AutoModelForCausalLM 3tokenizer_type: AutoTokenizer 4trust_remote_code: true 5is_llama_derived_model: true 6 7# huggingface repo 8datasets: 9 - path: teknium/OpenHermes-2.5 10 type: sharegpt 11 conversation: chatml 12 train_on_split: train 13 14 - path: abhinand/ultrachat_200k_sharegpt 15 type: sharegpt 16 conversation: chatml 17 train_on_split: train 18 19load_in_4bit: false 20load_in_8bit: false 21bf16: true # require >=ampere 22chat_template: chatml 23 24dataset_prepared_path: last_run_prepared_path 25hub_model_id: abhinand/TinyLlama-1.1B-OpenHermes-2.5-Chat-v1.0 26group_by_length: false 27 28val_set_size: 0.0 29sequence_len: 2048 30sample_packing: true 31pad_to_sequence_len: true 32 33adapter: lora 34lora_model_dir: 35lora_r: 32 36lora_alpha: 16 37lora_target_modules: 38 - q_proj 39 - v_proj 40 - k_proj 41 - o_proj 42 - gate_proj 43 - down_proj 44 - up_proj 45lora_modules_to_save: 46 - embed_tokens 47 - lm_head 48lora_dropout: 0.05 49lora_target_linear: true 50lora_fan_in_fan_out: 51 52output_dir: /home/tiny-llama/trained_models 53 54gradient_accumulation_steps: 2 55micro_batch_size: 32 56eval_batch_size: 32 57num_epochs: 1 58logging_steps: 1 59save_steps: 50 60save_total_limit: 3 61 62save_safetensors: true 63gradient_checkpointing: true 64 65lr_scheduler: cosine 66optimizer: "adamw_bnb_8bit" 67adam_beta2: 0.95 68adam_epsilon: 0.00001 69weight_decay: 0.1 70learning_rate: 0.0005 71max_grad_norm: 1.0 72warmup_ratio: 0.05 73# warmup_steps: 100 74 75flash_attention: true 76 77# Resume from a specific checkpoint dir 78resume_from_checkpoint: 79# If resume_from_checkpoint isn't set and you simply want it to start where it left off. 80# Be careful with this being turned on between different models. 81# auto_resume_from_checkpoints: true 82 83# wandb configuration if you're using it 84# Make sure your `WANDB_API_KEY` environment variable is set (recommended) or you login to wandb with `wandb login`. 85wandb_mode: # "offline" to save run metadata locally and not sync to the server, "disabled" to turn off wandb 86wandb_project: "tiny-llama-sft" 87wandb_name: 88wandb_run_id: 89 90special_tokens: 91 bos_token: "<s>" 92 eos_token: "</s>" 93 unk_token: "<unk>" 94tokens: # these are delimiters 95 - "<|im_start|>" 96 - "<|im_end|>" 97

Training procedure

Training hyperparameters

• learning_rate: 0.0005
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-05
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 476
• num_epochs: 1

Framework versions

• PEFT 0.8.2
• Transformers 4.38.0.dev0
• Pytorch 2.0.1
• Datasets 2.16.1
• Tokenizers 0.15.0

Open LLM Leaderboard Evaluation Results