See axolotl config

axolotl version: 0.4.1

yaml

1base_model: microsoft/phi-1_5 2model_type: AutoModelForCausalLM 3tokenizer_type: AutoTokenizer 4 5load_in_8bit: false 6load_in_4bit: true 7strict: false 8 9datasets: 10 - path: ptoro/honkers-phi 11 type: alpaca 12 13dataset_prepared_path: 14val_set_size: 0.05 15output_dir: ./outputs/phi-sft-out 16 17sequence_len: 2048 18sample_packing: true 19pad_to_sequence_len: true 20 21adapter: qlora 22lora_model_dir: 23lora_r: 64 24lora_alpha: 32 25lora_dropout: 0.05 26lora_target_linear: true 27lora_fan_in_fan_out: 28 29wandb_project: 30wandb_entity: 31wandb_watch: 32wandb_name: 33wandb_log_model: 34 35gradient_accumulation_steps: 1 36micro_batch_size: 2 37num_epochs: 4 38optimizer: adamw_torch 39adam_beta2: 0.95 40adam_epsilon: 0.00001 41max_grad_norm: 1.0 42lr_scheduler: cosine 43learning_rate: 0.000003 44 45train_on_inputs: false 46group_by_length: false 47bf16: auto 48fp16: 49tf32: true 50 51gradient_checkpointing: true 52gradient_checkpointing_kwargs: 53 use_reentrant: True 54early_stopping_patience: 55resume_from_checkpoint: 56local_rank: 57logging_steps: 1 58xformers_attention: 59flash_attention: true 60 61warmup_steps: 100 62evals_per_epoch: 4 63saves_per_epoch: 1 64debug: 65deepspeed: 66weight_decay: 0.1 67fsdp: 68fsdp_config: 69resize_token_embeddings_to_32x: true 70special_tokens: 71 pad_token: "<|endoftext|>" 72

outputs/phi-sft-out

• Loss: 1.5482

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 3e-06
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-05
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• num_epochs: 4

Training results

Framework versions

• PEFT 0.11.2.dev0
• Transformers 4.41.1
• Pytorch 2.1.2+cu118
• Datasets 2.19.1
• Tokenizers 0.19.1