See axolotl config

axolotl version: 0.4.1

yaml

1base_model: EleutherAI/pythia-160m-deduped 2load_in_8bit: false 3datasets: 4 - path: teknium/GPT4-LLM-Cleaned 5 type: alpaca 6dataset_prepared_path: 7val_set_size: 0.05 8adapter: lora 9lora_model_dir: 10sequence_len: 512 11lora_r: 16 12lora_alpha: 32 13lora_dropout: 0.05 14lora_target_modules: 15 - query_key_value 16 - dense 17 - dense_h_to_4h 18 - dense_4h_to_h 19lora_target_linear: 20lora_fan_in_fan_out: true # pythia/GPTNeoX lora specific 21output_dir: ./outputs/lora-alpaca-pythia 22gradient_accumulation_steps: 1 23micro_batch_size: 16 24num_epochs: 4 25learning_rate: 0.000005 26train_on_inputs: false 27group_by_length: false 28bf16: false 29tf32: true 30early_stopping_patience: 31resume_from_checkpoint: 32local_rank: 33weight_decay: 0.1 34evals_per_epoch: 4 35logging_steps: 1 36push_to_hub: tommyp111/pythia-160m-deduped-alpaca-lora 37wandb_project: pythia-alpaca-lora 38wandb_name: pythia-160m-grad-norm 39 40optimizer: adamw_torch 41adam_beta2: 0.95 42adam_epsilon: 0.00001 43max_grad_norm: 1.0 44gradient_checkpointing: true 45warmup_steps: 10000 46

outputs/lora-alpaca-pythia

• Loss: 2.2758

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 5e-06
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-05
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10000
• num_epochs: 4

Training results

Framework versions

• PEFT 0.12.0
• Transformers 4.44.2
• Pytorch 2.3.1+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1